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FIGURES 


Figure  1.  The  OPTEMA  sensor  eleetronies  inelude  Transmitter/Reeeiver  boards,  analog-to- 

digital  data  aequisition  hardware,  embedded  eontroller,  inverter,  and  operator  PC . 3 

Figure  2.  Map  view  diagram  of  the  OPTEMA  sensor  head  eonfiguration.  Horizontal  axis  (x’- 
and  y’-)  transmitters  are  eleetrieally  eonneeted  in  pairs  (x’-  with  x’-  and  y’-  with  y’-)  to 
provide  two  orthogonal  exeitation  axes.  The  larger  base  vertieal  axis  (z-)  transmitter 
provides  the  third  excitation  axis.  Blue  squares  represent  the  location  of  the  3-axis 
receivers . 4 

Figure  3.  OPTEMA  sensor  head  assembly  on  tow  frame  (left)  and  complete  system  in  operation 
(right) . 5 

Figure  4.  OPTEMA  library  data  for  an  81mm  mortar.  Magenta  lines  correspond  to  principal 
polarizabilities  recovered  from  static  data  files  associated  with  6  different  target 
orientations/standoffs.  Dark  grey  lines  correspond  to  the  averaged  library  polarizabilities..  6 

Figure  5.  Eibrary  polarizabilities  recovered  from  OPTEMA  data  (dark  grey  lines)  and 

MetalMapper  data  (light  grey  lines).  Polarizabilities  correspond  to  a  37mm  projectile  (left) 
and  a  small  ISO  (right).  The  slight  difference  in  late  time  decays  between  the  OPTEMA 
and  MetalMapper  polarizabilities  corresponding  to  the  ISO  is  likely  due  to  variations  in 


material  permeability  (different  ISO’s  were  used) . 7 

Figure  6.  OPTEMA  system  in  dynamic  survey  mode . 8 


Figure  7.  OPTEMA  Z-  component  (Transmit  Z-,  Receive  Z-)  detection  map  generated  from  test 
lane  data.  The  map  is  generated  by  summing  the  values  of  the  time  gates  between  400  and 
500  microseconds  in  each  Z-  component  data  channel.  This  lane  contains  three  targets  (ISO 
medium,  81  mm  mortar,  and  37mm  projectile)  all  buried  at  15  cm  depth.  ROIs 
corresponding  to  each  target  are  circled  in  red . 9 

Figure  8.  Polarizabilities  recovered  from  inversion  of  the  first  ROT  Blue,  red,  and  green  lines 
represent  the  primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from 
the  ROI  dynamic  data.  Grey  lines  represent  library  polarizabilities  generated  from  static 
calibration  data.  The  results  indicate  a  good  match  to  the  37mm  library . 10 

Figure  9.  Polarizabilities  recovered  from  inversion  of  the  second  ROI.  Blue,  red,  and  green  lines 
represent  the  primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from 
the  ROI  dynamic  data.  Grey  lines  represent  library  polarizabilities  generated  from  static 
calibration  data.  The  results  indicate  a  good  match  to  the  81mm  library . 10 

Figure  10.  Polarizabilities  recovered  from  inversion  of  the  third  ROI.  Blue,  red,  and  green  lines 
represent  the  primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from 
the  ROI  dynamic  data.  Grey  lines  represent  library  polarizabilities  generated  from  static 
calibration  data.  The  results  indicate  a  good  match  to  the  ISO  medium  library . 1 1 

Figure  11.  Diagram  of  APG  UXO  Technology  Demonstration  Site  showing  areas  surveyed.  The 
OPTEMA  demonstration  was  conducted  in  the  calibration  grid,  blind  grid,  and  indirect  fire 
areas . 19 
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Figure  12.  OPTEMA  survey  in  the  indireet  fire  area.  Exeessive  levels  of  standing  water  due  to 
reeent  rains  and  spring  thaw  produeed  several  exelusion  areas.  Overall,  approximately  70% 


eoverage  was  achieved  in  the  indirect  fire  area . 20 

Figure  13.  OPTEMA  transmitter  layout  and  dimensions . 22 

Eigure  14.  OPTEMA  sensor  head  receiver  cube  layout.  Receiver  cubes  are  10cm . 23 

Eigure  15.  OPTEMA  tow  sled . 24 


Figure  16.  Waveform  corresponding  to  3  OPTEMA  data  blocks.  Each  block  comprises  one 
sequence  of  Transmit-Z,  Transmit-Y,  or  Transmit-X.  The  number  of  base  waveform 
repeats  for  each  transmitter  cycle  depends  on  the  decay  period  selected.  This  example 
shows  an  8.3ms  decay  period.  This  period  produces  only  1  repeat  for  each  transmitter 
cycle.  If  stacking  is  increased,  this  complete  sequence  (three  blocks)  is  repeated  and  the 


result  is  averaged . 25 

Eigure  17.  Example  of  instrument  noise  standard  deviation  for  seven  receiver  cubes.  The  red 
line  represents  a  baseline  quality  objective  threshold.  The  magenta,  cyan  and  blue  lines 
correspond  to  the  z-,  y-,  and  x-  component  receiver  data . 26 

Eigure  18.  Calibration  ball  spike  test.  A  calibration  ball  is  placed  over  each  receiver  cube  in  a 
repeatable  location  to  identify  any  inconsistencies  in  data  channel  output . 27 


Eigure  19.  Calibration  ball  response  in  principal  Tx-Rx  pairings  for  seven  receiver  cubes.  The 
red  line  indicates  the  reference  measurement;  the  blue  line  indicates  the  measured  response. 
. 27 

Eigure  20.  Polarizabilities  recovered  from  daily  IVS  data  matched  to  a  library  reference.  Red, 
black  and  blue  lines:  primary,  secondary  and  tertiary  recovered  polarizabilities.  Grey  lines: 
reference  polarizabilities.  IVS  items  shown  here  are  medium  ISOs.  All  fits  are  within  95% 
match . 28 

Figure  21.  OPTEMA  data  map  (400  -  500  microsecond  time  gate  sum  of  Transmit  Z-/Receive  Z- 
channels)  of  the  calibration  grid . 29 

Figure  22.  Example  X-  (left)  ,  Y-  (center)  ,  and  Z-  (right)  detection  data  maps.  This  example  is 
for  a  37  mm  projectile  buried  at  15  cm  depth  lying  horizontally  across  the  data  collection 
swath . 32 

Eigure  23.  Optimal  target  locations.  Data  are  selected  for  soundings  acquired  while  the  sensor 
array  was  positioned  such  that  the  target  was  located  beneath  one  of  the  highlighted  regions. 
These  regions  correspond  to  areas  of  greater  orthogonality  between  transmitter  fields . 33 

Eigure  24.  TOP  LETT:  OPTEMA  DGM  Map  showing  several  anomalies  in  a  portion  of  the 
survey  area.  The  anomaly  of  interest  is  circled  in  black  with  the  model-based  estimated 
location  shown  as  the  magenta  dot.  TOP  RIGHT:  Library  matching  for  six  different  UXO 
targets.  Soundings  corresponding  to  the  anomaly  were  selected  for  inversion  and  the 
resulting  nine  sets  of  polarizabilities  (each  set  comprises  a  primary,  secondary,  and  tertiary 
polarizability)  are  plotted  in  magenta  against  the  TOl  library  polarizabilities  (shown  in  dark 
grey).  The  polarizabilities  obtained  from  these  consecutive  soundings  are  almost  identical 
and  show  a  clear  match  to  the  105mm  Indirect  Eire  munition  type.  BOTTOM:  Estimated 
model  parameters  corresponding  to  each  sounding.  Parameters  are  highly  consistent  for 
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each  sounding,  indicating  a  high  confidence  decision  can  be  made.  The  Northing  and 
Easting  errors  for  each  sounding  are  within  +/-  3  cm  of  the  mean  estimated  location  (shown 
as  the  magenta  dot  in  the  DGM  map).  Depth  estimates  are  consistent  to  within  +/-  2cm  and 
orientation  estimates  are  consistent  within  +/-  6  degrees . 34 

Figure  25.  Low  SNR  source.  Each  polarizability  (magenta  curves)  recovered  from  this  particular 
anomaly  source  is  noisy  as  a  result  of  the  object’s  depth;  however,  plotting  all  sets  of 
polarizabilities  recovered  from  soundings  within  the  ROI  against  the  relevant  libraries  (dark 


grey  curves)  makes  it  fairly  easy  to  identify  a  match  to  the  25mm  TOI . 35 

Figure  26.  Example  APG  scoring  template . 36 


Figure  27.  OPTEMA  ROC  curves  for  blind  grid  performance.  The  orange  dashed  line  shows 
probability  of  detection  vs.  probability  of  false  positive  for  ranking  assignments  based 
purely  on  sensor  response  amplitude  (response).  The  blue  solid  line  shows  probability  of 
detection  vs.  probability  of  false  positive  for  ranking  assignments  based  on  classification 
decisions  (discrimination).  The  black  solid  line  indicates  our  stop  dig  threshold  (the  point 
on  our  list  at  which  we  changed  assignment  from  ordnance  to  clutter).  The  black  dashed 
line  indicates  the  sensor  noise  threshold  (the  threshold  below  which  we  did  not  assign  a 
detection).  These  ROC  curves  indicate  that  we  can  achieve  detection  of  100%  of  the 
ordnance  with  85%  clutter  rejection.  For  this  analysis,  our  operating  threshold  was  set  with 
maximum  (100%)  efficiency  (i.e.,  the  point  where  maximum  clutter  rejection  can  be 
achieved  with  no  decrease  in  detection) . 37 

Figure  28.  OPTEMA  ROC  curves  for  open  field  indirect  fire  performance.  The  orange  dashed 
line  shows  probability  of  detection  vs.  probability  of  false  positive  for  ranking  assignments 
based  purely  on  sensor  response  amplitude  (response).  The  blue  solid  line  shows 
probability  of  detection  vs.  probability  of  false  positive  for  ranking  assignments  based  on 
classification  decisions  (discrimination).  The  black  solid  line  indicates  our  stop  dig 
threshold  (the  point  on  our  list  at  which  we  changed  assignment  from  ordnance  to  clutter). 
The  black  dashed  line  indicates  the  sensor  noise  threshold  (the  threshold  below  which  we 
did  not  assign  a  detection).  These  ROC  curves  indicate  that  we  can  achieve  detection  of 
99%  of  the  ordnance  with  83%  clutter  rejection  (based  on  100%  efficiency  operating 
threshold).  For  this  analysis,  our  operating  threshold  was  set  with  99%  efficiency . 38 

Figure  29.  LEFT:  OPTEMA  DGM  data  map  showing  the  anomaly  of  interest  circled  in  black. 
Using  the  2-D  data,  it  is  difficult  to  resolve  the  location  of  the  sources;  however, 
classification  analysis  of  the  data  reveals  two  distinct  sources.  RIGHT:  Polarizabilities  (red 
and  magenta  curves)  generated  from  classification  analysis  of  the  data  plotted  against 
library  polarizabilities  (dark  grey  curves).  Soundings  from  within  the  ROI  (circled  in  black) 
are  inverted  for  classification  features.  One  set  of  soundings  clearly  indicates  the  presence 
of  an  81mm  TOI  (red  polarizability  curves  corresponding  to  a  source  location  indicated  by 
the  red  dot  on  the  map)  while  another  set  of  soundings  indicates  a  large  piece  of  clutter 
(magenta  polarizability  curves  corresponding  to  a  source  location  indicated  by  the  magenta 
dot  on  the  map) . 42 

Figure  30.  LEFT:  DGM  map  showing  the  ROI  circled  in  black.  This  ROI  corresponds  to  a 

cluster  of  targets.  Using  the  threshold  target  picking  algorithm,  the  peak  detector  generates 
eight  potential  source  locations  in  the  ROI.  If  a  follow-up  cued  survey  were  used,  it  would 
entail  soundings  at  each  of  these  eight  locations,  requiring  several  minutes  to  complete. 
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RIGHT:  Polarizabilities  (red,  blue,  and  magenta  eurves)  generated  from  elassification 
analysis  of  the  data  plotted  against  library  polarizabilities  (dark  grey  curves).  The 
polarizabilities  are  divided  into  three  groups  (red,  blue,  magenta),  each  group  corresponding 
to  the  location  of  a  distinct  clutter  object  in  the  DGM  map  (red,  blue,  magenta  dots) . 42 
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EXECUTIVE  SUMMARY 


The  primary  objective  of  this  demonstration  was  to  assess  the  feasibility  of  acquiring 
classification  quality  data  during  dynamic  Electromagnetic  Induction  (EMI)  surveys  of 
Munitions  Response  Sites  (MRS).  The  process  of  obtaining  both  detection  and  classification 
data  during  a  single  dynamic  survey  is  known  as  the  one-pass  or  dynamic  classification  method 
of  data  collection.  By  demonstrating  the  dynamic  classification  methodology  using  the  One  Pass 
Time-domain  EMI  Array  (OPTEMA)  to  survey  areas  of  the  Aberdeen  Proving  Ground  (APG) 
Standardized  UXO  Technology  Demonstration  Site,  we  acquired  a  baseline  data  set  that  allowed 
us  to  quantify  the  detection  and  classification  performance  enabled  by  this  methodology  with 
performance  metrics  (e.g.,  probability  of  detection,  probability  of  false  alarm,  etc.) 
corresponding  to  a  standard  set  of  test  items.  Collecting  OPTEMA  data  in  the  calibration,  blind 
grid,  and  indirect  fire  areas  of  the  APG  UXO  site  provided  a  benchmark  to  gauge  system 
performance  as  these  areas  have  served  as  proving  grounds  for  a  large  number  of  UXO 
classification  sensors.  The  intent  of  this  demonstration  was  to  determine  if  the  dynamic 
classification  approach  can  enable  detection  and  classification  performance  comparable  to  that  of 
the  existing  set  of  cued  classification  sensors.  If  so,  this  performance  would  represent  a 
potentially  significant  improvement  in  operational  efficiency  for  classification  surveys. 

Overall,  the  objectives  for  the  demonstration  were  met.  Independent  scoring  of  the 
demonstration  results  indicated  that  the  system  enabled  detection  of  100%  of  the  ordnance  in  the 
blind  grid  with  a  false  positive  rejection  rate  of  85%,  and  detection  of  99%  of  the  ordnance  in  the 
open  field  indirect  fire  area  with  a  false  positive  rejection  rate  of  83%.  These  results  were 
achieved  with  a  production  rate  of  0.5acre/hr  in  the  blind  grid  and  0.3acre/hr  in  the  open  field. 
Production  rate  in  the  open  field  was  reduced  considerably  due  to  the  excessive  levels  of 
standing  water  throughout  the  site.  Saturated  ground  and  deep  water  holes  caused  several  delays 
due  to  tow  vehicle  immobilization  and  maneuvering  of  the  sensor  array  around  several  of  the 
water  holes.  Additionally,  these  site  conditions  resulted  in  total  survey  coverage  of 
approximately  70%  of  the  indirect  fire  area. 

During  the  analysis  of  our  demonstration  data,  the  benefit  of  being  able  to  directly  correlate 
classification  features  with  2-D  map  features  was  exceedingly  evident.  Using  the  same  high 
resolution  data  for  mapping  and  classification  makes  it  possible  to  reference  classification 
features  to  a  specific  set  of  coordinates  on  the  2-D  anomaly  map.  This  correlation  enables 
improved  characterization  of  the  2-D  target  space  in  high  anomaly  density  areas,  which  are 
particularly  challenging  for  cued  sensors  due  to  the  high  probability  of  cued  sensor  placement 
errors  and  the  subsequent  necessity  for  multiple  recollects.  Ultimately  the  ability  to  use  one  data 
set  for  target  detection  and  classification  could  lead  to  higher  confidence  in  classification 
decisions  for  these  high  density  survey  areas. 
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1.0  INTRODUCTION 


This  project  was  undertaken  by  White  River  Technologies,  Inc.  (WRT)  to  assess  the  feasibility 
of  acquiring  classification  quality  data  during  dynamic  Electromagnetic  Induction  (EMI)  surveys 
of  Munitions  Response  Sites  (MRS).  The  process  of  obtaining  both  detection  and  classification 
data  during  a  single  dynamic  survey  is  known  as  the  one-pass  or  dynamic  classification  method 
of  data  collection.  Currently,  standard  practice  for  classification  surveys  incorporates  a  two-step 
data  collection  process  where  a  preliminary  detection-level  survey  is  performed  as  part  of  the 
Digital  Geophysical  Mapping  (DGM)  operations.  The  DGM  data  serve  as  a  basis  for  identifying 
anomalies  (i.e.,  target  picking)  that  are  subsequently  interrogated  during  a  secondary  survey  by 
static  (or  cued)  EMI  measurements. 

Commercial  sensors  such  as  the  Geometries  MetalMapper  are  configured  for  cued  surveys  and, 
accordingly,  have  demonstrated  significant  success  when  applied  in  this  two-step  process.  While 
this  cued  mode  is  extremely  effective  for  distinguishing  Munitions  and  Explosives  of  Concern 
(MEC)  items  from  non-hazardous  objects,  it  has  two  significant  drawbacks.  Eirst,  incorporating 
a  secondary  cued  EMI  survey  creates  additional  costs.  While  the  cued  survey  is  more  efficient 
than  excavation  of  all  anomalies,  it  is  typically  more  time  consuming  than  the  preliminary  DGM 
survey  and  thus  adds  significantly  to  the  survey  costs  allocated  for  the  project.  Second,  because 
the  DGM  and  cued  portions  of  the  survey  apply  different  sensors  (e.g.,  Geonics  EM61  and 
MetalMapper),  there  is  often  a  disconnect  between  the  target  picking  process  and  the  cued 
survey  process.  The  lower  spatial  and  temporal  resolution  of  the  DGM  sensor  can  lead  to 
misguided  target  picking  in  high  density  areas. 

The  one-pass  method  offers  improvements  over  the  cued  method  in  regards  to  these  limitations. 
Detection  and  classification  achieved  from  a  single  EMI  survey  is  efficient  and  could 
significantly  reduce  the  costs  associated  with  the  burden  of  an  additional  cued  survey. 
Eurthermore,  using  high  resolution  classification  sensors  for  detection  and  identification  of 
anomalies  could  ultimately  provide  better  initial  characterization  of  the  target  space  than  that 
typically  afforded  by  the  standard  DGM  sensors.  This  improved  characterization  could  lead  to 
better  classification  results  in  high  density  areas. 

In  conducting  this  project,  our  primary  objective  is  to  determine  whether  it  is  possible,  using  a 
one-pass  survey  mode,  to  achieve  classification  results  comparable  to  those  achieved  during  cued 
surveys.  Because  of  the  anticipated  lower  operating  costs  associated  with  dynamic  classification 
surveys,  comparable  classification  quality  would  indicate  that  one-pass  survey  modes  may  offer 
a  better  solution  than  cued  surveys  for  many  projects. 

1.1  BACKGROUND 

The  practical  basis  for  munitions  classification  is  founded  on  the  development  of  advanced  EMI 
sensors  that  provide  high  spatial  and  temporal  resolution  data.  The  data  produced  by  these 
sensors  enable  the  application  of  advanced  physical  models  to  extract  useful  classification 
parameters  that  correspond  to  physical  properties  of  the  object.  The  initial  intent  of  these 
classification  technologies  was  to  provide  a  means  for  inserting  additional  steps  into  the  existing 
data  collection  and  processing  workflows  for  Munitions  Response  (MR)  projects  without  altering 
the  established  protocols.  Accordingly,  the  cued  survey  process  was  developed  as  an  add-on  to 
the  existing  geophysical  survey  workflow  for  such  projects.  By  relying  on  the  standard  DGM 
survey  data  for  target  picking,  advanced  sensors  could  be  incorporated  in  cued  mode  into  the 
existing  process  with  minimal  impact  to  the  overall  flow.  This  process  has  demonstrated 
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significant  success  for  discriminating  Targets  of  Interest  (TOI)  from  elutter  at  demonstration  and 
produetion  sites  [1]. 

With  the  gradual  trend  towards  aeeeptanee  of  classification  technologies  in  the  produetion 
environment,  the  possibility  now  exists  for  shifting  the  foeus  of  elassifieation  teehnology 
development  from  improved  performance  to  improved  effieieney  and  feasibility.  One-pass 
deteetion  and  elassifieation  may  provide  the  greatest  return  on  investment  for  future 
elassifieation  technology  development.  One-pass  surveys  eould  effectively  reduce  exeavation 
eosts  without  inereasing  geophysieal  survey  eosts  to  the  extent  required  for  cued  sensor  data 
eolleetion.  While  one-pass  survey  eosts  may  be  somewhat  higher  than  those  of  a  standard  DGM 
survey  as  a  result  of  slightly  lower  survey  speeds,  this  inerease  would  be  more  than  offset  by 
eliminating  the  eosts  assoeiated  with  the  follow-up  eued  survey. 

Given  the  reeent  suceesses  of  eued  EMI  sensors  and  the  potential  benefits  of  dynamie 
elassifieation,  several  demonstrations  have  been  performed  to  evaluate  the  effeetiveness  of 
applying  eued  EMI  sensors,  sueh  as  the  TEMTADS  2x2  variant  and  MetalMapper  advaneed 
sensors,  in  a  dynamie  mode. 

We  developed  the  One-Pass  Time-Domain  EMI  Array  (OPTEMA)  for  the  speeific  purpose  of 
aequiring  elassifieation-level  data  in  a  dynamie  survey  mode  rather  than  a  eued  survey  mode. 
Consequently,  the  OPTEMA  eomprises  certain  features  that  are  designed  to  produee  high  quality 
elassifieation  data  during  dynamie  surveys.  Details  of  these  design  elements  are  presented  in 
Seetion  2. 

1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

The  primary  objective  of  this  demonstration  was  to  quantify  the  deteetion  and  elassifieation 
performanee  of  the  OPTEMA  sensor.  By  demonstrating  the  system  at  the  Aberdeen  Proving 
Ground  (APG)  Standardized  UXO  Teehnology  Demonstration  Site,  we  aequired  a  baseline  data 
set  that  enabled  us  to  quantify  the  deteetion  and  elassifieation  performanee  of  the  OPTEMA  with 
performance  metries  (e.g.,  probability  of  detection,  probability  of  false  alarm,  ete.) 
eorresponding  to  a  standard  set  of  test  items.  Colleeting  OPTEMA  data  in  the  ealibration,  blind 
grid,  and  indireet  fire  areas  of  the  APG  UXO  site  provided  a  benehmark  to  gauge  system 
performance  as  these  areas  have  served  as  proving  grounds  for  a  large  number  of  UXO 
elassifieation  sensors.  The  intent  of  this  demonstration  was  to  determine  if  the  OPTEMA  ean 
provide,  in  dynamie  operation  mode,  deteetion  and  elassifieation  performanee  eomparable  to  that 
of  the  existing  set  of  eued  elassifieation  sensors.  If  so,  this  performanee  would  represent  a 
potentially  significant  improvement  in  operational  effieieney  for  classification  surveys. 

1.3  REGULATORY  DRIVERS 

Demonstration  of  this  technology  will  elucidate  potential  eost  savings  to  munitions  remediation 
projeets  as  a  result  of  deereased  time  and  labor  devoted  to  elassifieation  surveys.  DoD  direetives 
for  munitions  response  projects  now  include  seope  for  elassifieation  teehnologies  at  a  number  of 
sites.  We  expect  that  over  the  next  5-10  years,  the  list  of  sites  amenable  to  classification 
practices  will  inerease  signifieantly.  Consequently,  any  teehnologies  that  streamline  operations 
assoeiated  with  munitions  elassifieation  eould  have  a  large  impaet  on  the  DoD’s  ability  to 
effeetively  implement  these  teehnologies.  By  replacing,  to  the  extent  possible,  the  DGM/cued 
survey  sequenee  with  a  one-pass  detection  and  classification  survey,  it  may  be  possible  to  extend 
the  feasibility  of  applying  elassifieation  to  a  broader  range  of  sites. 
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2.0  TECHNOLOGY 


The  OPTEMA  comprises  an  array  of  multi-directional  transmitters  and  receivers  that  are 
optimally  configured  to  provide  good  EMI  characterization  across  the  entire  sensor  swath.  This 
capability  is  the  basis  for  effective  dynamic  classification  since  sensor  position  during  dynamic 
surveys  is  based  on  survey  transects  rather  than  on  a  priori  target  location.  Consequently,  it  is 
likely  that  a  large  number  of  targets  will  be  located  at  some  lateral  offset  relative  to  the  array 
center  during  a  dynamic  survey.  Details  of  the  OPTEMA  design  and  operation  are  presented  in 
the  following  subsections. 

2.1  TECHNOLOGY  DESCRIPTION 


The  OPTEMA  sensor  is  built  around  the  G&G  Sciences  National  Instruments  (Nl)-based  data 
acquisition  framework.  These  data  acquisition  components  are  similar  to  those  incorporated  in 
the  first  generation  commercial  MetalMapper  systems.  Data  acquisition  hardware  is  housed  in  a 
National  Instruments  PXI-I042  chassis  and  includes  an  NI  PXI-8108  embedded  controller 
(Windows  OS)  and  six  8-channel  16-bit  NI  PXI-6I43  A/D  cards.  Intermediate  hardware  is 
housed  in  an  external  module  and  includes  the  transmitter  controller  and  power  distribution 
board,  and  three  16-channel  receiver  boards.  These  components  along  with  a  2000  Watt  inverter 
are  contained  in  a  ruggedized  vehicle-mounted  chassis  (Eigure  1)  and  compose  the  OPTEMA 
sensor  electronics. 


Tx/Rx  Module 

NI  PX 1-1042 
Chassis 

Inverter 


Operator  PC 


Figure  1.  The  OPTEMA  sensor  electronics  include  Transmitter/Receiver  boards,  analog-to-digital  data 
acquisition  hardware,  embedded  controller,  inverter,  and  operator  PC. 
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One  of  the  key  differences  between  the  OPTEMA  and  MetalMapper  data  acquisition 
components  is  the  effective  doubling  in  number  of  receiver  channels  (OPTEMA  provides  42 
receiver  channels  compared  to  the  MetalMapper’s  21  channels).  The  increase  in  receiver 
channels  led  to  the  addition  of  three  PXI-6143  cards  in  the  hardware  design.  Additionally,  a 
modified  version  of  the  MetalMapper  data  acquisition  software  (EMSDAcquire)  is  needed  to 
manage  the  additional  data  channels. 

The  OPTEMA  sensor  head  comprises  five  transmitters  and  fourteen  3 -axis  receivers  across  a  1.8 
meter  sensor  swath  (Eigure  2).  This  design  ensures  that  three  orthogonal  magnetic  fields  are 
produced  at  any  across  track  location.  The  distribution  of  the  14  receivers  also  ensures  that 
fields  scattered  by  any  target  located  across  the  sensor  swath  will  be  characterized  sufficiently  to 
invert  the  data  and  constrain  target  classification  parameters. 


1.0  m 


<- 


1.8  m 


-> 


Figure  2.  Map  view  diagram  of  the  OPTEMA  sensor  head  configuration.  Horizontal  axis  (x’-  and  y’-) 
transmitters  are  electrically  connected  in  pairs  (x’-  with  x’-  and  y’-  with  y’-)  to  provide  two  orthogonal 
excitation  axes.  The  larger  base  vertical  axis  (z-)  transmitter  provides  the  third  excitation  axis.  Blue  squares 
represent  the  location  of  the  3-axis  receivers. 

The  transmitter  coils  include  four  horizontal  axis  transmitters  (x’-  and  y’-  axes)  and  one  large 
vertical  axis  transmitter  (z-  axis).  The  horizontal  axis  transmitters  are  connected  in  series  pairs 
(i.e.,  x’-  with  x’-  and  y’-  with  y’-)  to  provide  two  effective  orthogonal  excitation  axes.  The 
horizontal  axis  transmitters  and  triaxial  receivers  (aka,  receiver  ’cubes')  share  the  same  reference 
coordinate  frame  (primed)  that  is  rotated  45  degrees  from  the  principal  coordinate  system  (i.e., 
referenced  to  the  direction  of  travel). 

Sensor  array  position  and  orientation  data  are  provided  by  a  Trimble  RTK  DGPS  R8  receiver 
and  a  Microstrain  3DM-GX3-25  inertial  sensor,  respectively.  Position  and  orientation  sensors 
are  mounted  on  the  sensor  tow  frame,  which  provides  undercarriage  protection  and  mobility  for 
the  sensor  head.  Pictures  of  the  complete  sensor  head  and  tow  assembly  are  shown  in  Eigure  3. 
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Figure  3.  OPTEMA  sensor  head  assembly  on  tow  frame  (left)  and  complete  system  in  operation  (right). 


Each  OPTEMA  transmitter  produces  an  exponential  current  ramp  that  is  rapidly  terminated  to 
generate  a  strong  electromotive  force  in  nearby  targets.  The  receivers  measure  the  decay  of 
secondary  magnetic  field  contributions  over  time  gates  spanning  a  period  from  24  microseconds 
to  up  to  approximately  8  milliseconds.  The  combination  of  three  effective  transmitters  (including 
horizontal  axis  transmitters  in  pairs)  and  fourteen  3 -axis  receivers  provides  126  data  channels. 
The  magnetic  field  decays  measured  by  the  receivers  usually  include  contributions  from  system 
components  (e.g.,  supporting  hardware,  survey  vehicle,  etc.).  To  maximize  sensitivity  to 
anomalies,  a  background  response  is  typically  acquired  to  establish  these  intrinsic  sources,  and 
later  subtracted  from  subsequent  data  sets  to  isolate  the  anomaly  response.  In  addition  to  a 
background  subtraction,  a  transmitter  current  normalization  is  applied  to  all  data  files  to  account 
for  any  small  discrepancies  in  current  between  the  three  transmitters. 


The  OPTEMA  uses  a  50%  duty  cycle  60  Hz  sub-harmonic  as  the  base  waveform.  Data  blocks 
(of  duration  BlockT)  are  built  from  a  specified  number  of  repeats  (nRepeats)  of  this  base 
waveform.  The  base  waveform  comprises  a  single  period  of  a  50%  duty  cycle  square  wave 
voltage.  During  the  voltage-on  periods,  the  current  in  the  energized  transmitter  coil  ramps 
exponentially  according  to  the  coil  time  constant  parameters.  During  the  voltage-off  periods,  the 
receivers  measure  the  ambient  magnetic  field  decays.  These  decays  are  averaged  with  the 
appropriate  sign  changes  for  the  positive  and  negative  half  cycles  of  the  base  period.  Further 
signal  averaging  is  applied  for  each  repeat  of  the  base  waveform  within  the  data  block.  In 
addition  to  the  base  repeats,  the  operator  can  select  a  number  of  blocks  to  “stack”  (nStk)  or 
average,  in  order  to  improve  signal-to-noise  ratio.  Thus,  the  OPTEMA  provides  noise  reduction 
through  the  inherent  base  repeat  averaging  and  the  explicit  stack  averaging. 


Understanding  the  effect  of  these  data  acquisition  parameters  on  detection  and  classification 
performance  is  a  key  component  of  this  demonstration.  The  OPTEMA  can  be  operated  in  either 
static  or  dynamic  mode.  As  with  cued  EMI  sensors,  the  static  mode  is  ideal  for  maximizing 
signal-to-noise  ratio  (SNR).  Because  there  are  no  restrictions  on  sample  rate  when  the  sensor  is 
operated  in  static  mode,  data  stacking  can  be  maximized  to  reduce  noise.  Increasing  the  number 
of  samples  stacked  significantly  decreases  the  sample  rate.  Thus,  for  dynamic  mode  where 
higher  sample  rates  are  required  to  maximize  data  density,  stacking  must  be  reduced  as  much  as 
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possible  while  allowing  the  sensor  to  maintain  adequate  SNR  for  deteetion  and  classification. 
Table  1  provides  an  example  of  typical  data  acquisition  parameters  applied  during  static 
operation  of  the  OPTEMA. 

Table  1.  OPTEMA  static  data  acquisition  parameters. 


Mode 

Hold-Off 

Time 

(ins) 

BlockT 

(s) 

uRpts 

Window 

Width 

(%) 

nStks 

RxMode 

TxMode 

Static 

50 

0.033 

1 

10 

20 

Decay 

Decimated 

ZYX 

Notes: 

ps  =  microsecond 

mv  =  millivolts 

nRpts  =  number  of  repeats 

nStks  =  number  of  stacks 

%  =  percent 

s  =  second 

For  the  OPTEMA  system,  the  static  mode  is  used  primarily  for  collection  of  reference  library 
data.  Eibrary  data  sets  can  be  acquired  with  targets  placed  directly  beneath  the  sensor  (e.g.,  test 
pit  data  collection)  or  with  targets  placed  above  the  sensor  (e.g.,  test  jig  data  collection).  For 
library  static  mode,  stacks  are  increased  to  10  samples  to  maximize  SNR.  Targets  are  placed  in 
several  orientations  and  at  several  depths/heights  to  ensure  recovery  of  consistent  polarizability 
features  for  all  potential  target  encounters.  Once  a  sufficient  sample  of  orientations/depths  is 
achieved,  polarizability  features  recovered  from  inversion  of  each  static  file  are  averaged  to 
produce  a  library  set  of  polarizabilities  for  the  target.  Figure  4  shows  an  example  of  OPTEMA 
library  features  corresponding  to  an  81mm  mortar. 


Figure  4.  OPTEMA  library  data  for  au  81mm  mortar.  Mageuta  Hues  correspoud  to  priucipal  polarizabilities 
recovered  from  static  data  files  associated  with  6  differeut  target  orieutatious/staodoffs.  Dark  grey  Hues 
correspoud  to  the  averaged  library  polarizabilities. 

Polarizabilities  recovered  from  inversion  of  OPTEMA  data  are  comparable  to  features  recovered 
from  other  cued  EMI  sensors  such  as  the  MetalMapper.  Figure  5  shows  an  example  of  library 
data  collected  by  the  OPTEMA  and  MetalMapper  systems. 
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Figure  5.  Library  polarizabilities  recovered  from  OPTEMA  data  (dark  grey  lines)  and  MetalMapper  data 
(light  grey  lines).  Polarizabilities  correspond  to  a  37mm  projectile  (left)  and  a  small  ISO  (right).  The  slight 
difference  in  late  time  decays  between  the  OPTEMA  and  MetalMapper  polarizabilities  corresponding  to  the 
ISO  is  likely  due  to  variations  in  material  permeability  (different  ISO’s  were  used). 

With  the  exception  of  the  static  mode  used  for  library  data  collection  and  sensor  calibration,  the 
primary  data  collection  mode  for  the  OPTEMA  is  dynamic.  In  dynamic  mode  the  OPTEMA 
provides  both  detection  and  classification  information.  Several  data  acquisition  parameter 
options  are  possible  for  dynamic  survey  mode.  The  target  survey  speed  is  approximately  0.5 
m/s.  Because  the  nominal  data  block  size  (a  single  Transmit-Z,  Transmit-Y,  or  Transmit-X 
sequence)  is  33  ms,  this  advance  rate  results  in  1  OPTEMA  data  sample  (the  combined  Transmit- 
Z,  Transmit-Y,  Transmit-X  sequence)  every  5  cm  (assuming  only  1  sample  per  stack).  If 
multiple  stacks  are  used,  the  sampling  resolution  decreases  (for  example,  3  stacks  produces  1 
sample  every  15  cm).  Within  each  data  block,  multiple  waveform  cycles  may  be  averaged 
depending  on  the  decay  window  selected.  Eor  an  8.3  ms  decay  period,  only  a  single  bipolar 
waveform  is  acquired  for  each  transmitter  cycle  within  the  data  block.  If  the  decay  time  is 
changed  to  2.8  ms,  three  waveform  periods  are  acquired  for  each  transmitter  block,  thus 
producing  an  inherent  stacking  of  3  samples  (i.e.,  nRpt=3). 

The  primary  tradeoff  to  consider  when  selecting  dynamic  data  acquisition  parameters  is  between 
data  SNR  and  data  density  (i.e.,  spatial  resolution).  Increasing  the  number  of  stacks  will  provide 
higher  data  SNR,  but  will  compromise  spatial  sampling  resolution  both  of  which  are  factors  that 
influence  the  quality  of  classification  parameters.  Another  consideration  is  the  tradeoff 
associated  with  decay  period.  Inherent  stacking  (nRpts),  and  therefore  data  SNR,  can  be 
increased  by  shortening  the  decay  period  from  8.3  ms  (I  effective  stack)  to  2.8  ms  (3  effective 
stacks);  however,  the  shorter  decay  period  limits  the  ability  to  extract  late  time  parameters  from 
the  data,  features  that  are  particularly  important  for  classifying  larger  or  thick-walled  objects. 

Selection  of  optimal  dynamic  survey  parameters  represents  a  key  component  of  this 
demonstration.  Based  on  preliminary  performance  tests,  we  have  settled  on  four  sets  of  default 
data  acquisition  parameters  for  OPTEMA  surveys.  Table  2  lists  these  parameter  settings. 
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Table  2.  OPTEMA  default  dynamic  data  acquisition  parameter  settings. 


Setting 

Decay  (ms) 

nStk 

nRpt 

Total  Stacks 

Sample  Resolution  (cm) 

1 

8.3 

1 

1 

1 

5 

2 

8.3 

2 

1 

2 

10 

3 

2.8 

1 

3 

3 

5 

4 

2.8 

2 

3 

6 

10 

Once  dynamic  acquisition  parameters  are  set,  the  OPTEMA  is  ready  for  operation.  Line  files 
generated  for  the  survey  area  are  loaded  into  the  EMSDAcquire  program.  These  files  contain  the 
coordinates  of  pre-determined  transects  within  the  survey  area.  EM3DAcquire  provides  line 
following  based  on  the  sensor  head  GPS  and  IMU  data.  Using  the  line  following  display,  the 
OPTEMA  is  aligned  with  the  starting  coordinates  of  the  selected  transect.  Once  the  sensor  is 
sufficiently  aligned  (i.e.,  vehicle  and  sled  heading  is  approximately  in-line  with  transect),  the 
EMSDAcquire  “start  acquisition”  button  is  selected  and  the  program  will  start  recording  dynamic 
data.  Eigure  6  shows  an  image  of  the  OPTEMA  collecting  data  along  a  transect  in  our  test  field. 


Figure  6.  OPTEMA  system  in  dynamic  survey  mode. 


After  completing  a  transect  (or  transects)  the  OPTEMA  data  can  be  stitched  together  to  form  a 
detection  map.  Several  options  exist  for  detection.  Early,  middle,  or  late  time  gates  within  the 
Z-,  Y-,  or  X-  component  data  can  be  used  to  generate  a  detection  map.  Subsequently,  Regions  of 
Interest  (ROIs)  are  selected  from  the  map  based  on  a  detection  threshold  value  (determined  from 
noise  floor  standard  deviation).  Eigure  7  shows  a  detection  map  generated  from  data  collected 
over  one  of  our  test  lanes.  To  generate  detection  ROIs,  we  sum  the  values  of  the  time  gates 
between  400  and  500  microseconds  in  the  Transmit  Z-/Receive  Z-  data  channels. 
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Z-Component  Detection  Map 


Figure  7.  OPTEMA  Z-  component  (Transmit  Z-,  Receive  Z-)  detection  map  generated  from  test  lane  data. 
The  map  is  generated  by  summing  the  values  of  the  time  gates  between  400  and  500  microseconds  in  each  Z- 
component  data  channel.  This  lane  contains  three  targets  (ISO  medium,  81  mm  mortar,  and  37mm 
projectile)  all  buried  at  15  cm  depth.  ROIs  corresponding  to  each  target  are  circled  in  red. 

Once  threshold  analysis  is  complete;  classification  is  performed  on  each  ROI  extracted  from  the 
data  grid.  Eaeh  ROI  comprises  a  number  of  effective  statie  soundings.  Depending  on  the  target 
size,  target  depth,  and  data  sample  resolution  a  typical  ROI  will  comprise  25-50  useful 
soundings.  Thus,  a  very  large  volume  of  data  exists  for  classification. 

There  are  several  options  for  selecting  data  to  classify.  Inversion  may  be  performed  on  each 
individual  sounding  in  the  ROI  or  on  a  subset  of  soundings  within  the  ROI  and  the  resulting 
classification  features  ean  be  averaged  to  provide  one  set  of  polarizabilities  for  the  ROI. 
Alternatively,  data  from  multiple  soundings  within  the  ROI  ean  be  aggregated  and  an  inversion 
can  be  performed  on  the  resulting  eomposite  data  set.  This  approaeh  has  the  benefit  of  providing 
a  data  set  that  provides  optimal  constraint  on  the  inversion;  however,  its  effeetiveness  depends 
heavily  on  the  accuracy  of  the  position  and  orientation  data.  The  relative  (i.e.,  point-to-point) 
position  and  orientation  of  the  sensor  head  for  eaeh  sounding  must  be  incorporated  in  the 
forward  model  to  produce  effective  results.  Any  errors  in  the  relative  position  for  each  sounding 
can  lead  to  a  poor  model  fit  when  the  inversion  is  applied. 

Based  on  our  testing  to  date,  we  have  found  two  methods  that  produce  good  results.  Both 
methods  require  selecting  a  subset  of  soundings  to  use  for  inversion.  Soundings  are  selected 
based  on  the  number  of  receiver  channels  in  each  sounding  that  exceed  an  SNR  threshold  and  the 
across-track  position  of  the  anomaly  within  the  ROI  (across-track  position  of  the  anomaly 
determines  whieh  soundings  provide  the  best  orthogonal  magnetic  field  excitation).  Typically, 
3-10  soundings  within  each  ROI  will  meet  these  criteria.  Once  the  subset  of  soundings  is 
selected,  our  first  method  is  to  invert  the  data  in  each  sounding  separately  and  average  the 
results.  The  second  method  involves  aggregating  data  from  each  sounding  in  the  subset  and 
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inverting  the  composite  data  set.  Figure  8,  Figure  9,  and  Figure  10  show  the  results  of  applying 
the  first  method  (inversion  on  an  individual  basis  and  subsequent  averaging  of  the 
polarizabilities)  to  the  three  ROls  shown  in  Figure  7. 


Figure  8.  Polarizabilities  recovered  from  inversion  of  the  first  ROI.  Blue,  red,  and  green  lines  represent  the 
primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from  the  ROI  dynamic  data.  Grey 
lines  represent  library  polarizabilities  generated  from  static  calibration  data.  The  results  indicate  a  good 
match  to  the  37mm  library. 


Figure  9.  Polarizabilities  recovered  from  inversion  of  the  second  ROI.  Blue,  red,  and  green  lines  represent 
the  primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from  the  ROI  dynamic  data. 
Grey  lines  represent  library  polarizabilities  generated  from  static  calibration  data.  The  results  indicate  a 
good  match  to  the  81mm  library. 
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Figure  10.  Polarizabilities  recovered  from  inversion  of  the  third  ROI.  Blue,  red,  and  green  lines  represent  the 
primary,  secondary,  and  tertiary  (respectively)  polarizabilities  recovered  from  the  ROI  dynamic  data.  Grey 
lines  represent  library  polarizabilities  generated  from  static  calibration  data.  The  results  indicate  a  good 
match  to  the  ISO  medium  library. 

Another  key  eomponent  of  this  demonstration  is  identifying  the  optimal  methods  for  seleeting 
and  inverting  dynamic  data.  Prior  to  the  demonstration,  we  applied  the  aforementioned  methods 
successfully  on  test  data  acquired;  however,  applying  these  methods  to  a  larger  data  set  acquired 
during  the  demonstration  that  includes  more  target  diversity  will  further  elucidate  any 
refinements  to  these  methodologies. 

2.2  TECHNOLOGY  DEVELOPMENT 

Development  of  the  OPTEMA  technology  has  progressed  through  two  stages.  Under  project 
MM-0908,  the  initial  design  concept  for  a  one-pass  detection  and  classification  system  was 
evaluated  to  identify  optimal  hardware  components.  Initially,  this  system  concept  was  based  on 
modifications  to  the  Time  domain  ElectroMagnetic  Towed  Array  Detection  System 
(TEMTADS)  hardware,  which  comprised  mono-loop  (i.e.,  single-axis)  transmitter  and  receiver 
coils.  During  this  preliminary  phase  of  development,  early  proof-of-concept  tests  in  conjunction 
with  modeling  results  indicated  that  it  would  be  beneficial  to  use  three-axis  components  instead 
of  the  TEMTADS  coils.  Consequently,  a  first  prototype  OPTEMA  was  developed  using  a 
modified  version  of  the  Advanced  Ordnance  Locator  (AOL)  system  architecture. 

A  key  feature  of  this  first  generation  system  was  the  incorporation  of  the  Novatel  SPAN,  which 
integrates  RTK  GPS  with  a  high  performance  Inertial  Navigation  System  (INS)  to  produce 
extremely  high  precision/accuracy  position  and  orientation  data.  The  initial  development  stage 
(project  MM-0908)  culminated  with  a  demonstration  of  the  first  generation  OPTEMA  prototype 
at  the  Yuma  Proving  Ground  (YPG)  Standardized  UXO  Demonstration  Site  [2]. 

Under  the  current  demonstration  project  (MR-201225),  we  have  performed  several  additional 
modifications  to  the  earlier  prototype.  These  advances  are  related  primarily  to  improvements  in 


White  River  Technologies 
APG  Demonstration  Report 


11 


September  2014 


design  feasibility  and  funetionality  for  produetion  operations.  Speeifieally,  we  redesigned  the 
sensor  head  to  eover  a  wider  swath.  This  feature  required  the  addition  of  5  reeeiver  eubes  (from 
9  to  14)  and  2  transmitter  eoils  (from  3  to  5)  as  well  as  the  eomplete  reeonfiguration  of  the  sensor 
head  layout.  The  sensor  head  design  now  provides  thorough  multi-axis  illumination  and  enables 
high  spatial  resolution  sampling  aeross  the  entire  1.8m  sensor  swath.  The  inerease  in  reeeivers 
and  transmitters  also  required  several  modifieations  to  the  data  aequisition  system,  whieh 
ineluded  the  addition  of  several  A/D  boards  as  well  as  a  new  version  of  the  data  aequisition 
software. 

Another  key  differenee  in  this  next  generation  system  is  the  absenee  of  the  Novatel  SPAN. 
While  the  SPAN  provided  very  high  quality  position/orientation  data,  it  proved  to  be  eost 
prohibitive  from  a  feasibility  standpoint.  Aeeordingly,  this  eurrent  stage  of  development  has 
foeused  on  redueing  the  elassifieation  performanee  dependeney  on  extremely  high  aeeuraey 
position/orientation  data  sueh  that  standard  RTK  differential  GPS  eombined  with  a  low  eost 
inertial  sensor  will  suffiee  for  position/orientation  requirements.  We  have  tested  and 
implemented  the  aforementioned  data  proeessing  methods  to  ereate  an  approaeh  that  is  not 
reliant  on  the  highest  quality  position/orientation  data. 

A  timeline  summarizing  OPTEMA  system  development  is  provided: 

Initial  Design  Coneept  2009  (MM-0908):  modeling,  simulation,  and  benehtop  data 
eolleetion  to  identify  optimal  hardware  eomponents;  modified  design  eoneept  from 
TEMTADS  variant  to  AOE  variant; 

Positioning  Requirements  Assessment  2009  (MM-0908):  proof-of-eoneept  data 
eolleetion  to  assess  the  neeessity  for  high  quality  position/orientation  data;  resulted  in  the 
ineorporation  of  the  Novatel  SPAN  into  the  first  generation  prototype  design; 

Eirst  Generation  System  Demonstration  2010/2011  (MM-0908):  initial  development 
stage  eulminated  with  a  demonstration  of  the  AOE-based  system  at  YPG;  system 
ineluded  the  high  performanee  SPAN  during  this  demonstration; 

Next  Generation  Design  Evaluation  2012  (MR-1225):  design  and  fabrieation  of  eustom 
sensor  head  eomponents;  integration  of  eomponents  in  new  eonfiguration;  integration, 
testing,  and  development  of  new  data  aequisition  hardware  and  software; 

Data  Proeessing  Refinement  2013/2014  (MR- 1225):  evaluated  testbed  data  to  refine 
elassifieation  proeessing  methods  that  would  reduee  relianee  on  position  and  orientation 
data;  developed  software  proeessing  modules  that  ineorporated  these  methods  for 
proeessing  production  level  data; 

Next  Generation  System  Demonstration  2014  (MR-1225):  performed  dynamic 
classification  surveys  at  APG  demonstration  site  to  gauge  system  performance 
enhancements;  system  included  hardware  components  described  in  the  previous 
subsection. 

2,3  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  primary  advantage  of  the  OPTEMA  technology  over  existing  cued  EMI  sensors  is  the  ability 
to  acquire  effective  classification  data  in  dynamic  mode.  This  capability  means  that  the 
OPTEMA  delivers  detection  and  classification  in  a  DGM  survey  mode,  thus  eliminating  the  need 
for  a  secondary  cued  survey.  By  removing  the  cued  survey  from  the  workflow,  the  one-pass 
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method  not  only  improves  the  overall  effieiency  of  the  geophysieal  survey  proeess;  it  also 
removes  the  ambiguity  assoeiated  with  reeoneiling  high  resolution  (spatial  and  temporal)  eued 
EMI  data  with  lower  resolution  DGM  data. 

During  the  analysis  of  our  demonstration  data,  the  benefit  of  being  able  to  directly  correlate 
classification  features  with  2-D  map  features  was  exceedingly  evident.  Using  the  same  high 
resolution  data  for  mapping  and  classification  makes  it  possible  to  reference  classification 
features  to  a  specific  set  of  coordinates  on  the  2-D  anomaly  map.  This  correlation  enables 
improved  characterization  of  the  2-D  target  space  in  high  anomaly  density  areas,  which  are 
particularly  challenging  for  cued  sensors  due  to  the  high  probability  of  cued  sensor  placement 
errors  and  the  subsequent  necessity  for  multiple  recollects.  Ultimately  the  ability  to  use  one  data 
set  for  target  detection  and  classification  could  lead  to  higher  confidence  in  classification 
decisions  for  these  high  density  survey  areas. 

The  main  limitations  of  the  current  technology  are  associated  with  the  reduced  data  SNR 
compared  to  that  of  cued  sensors  such  as  the  MetalMapper.  There  are  two  reasons  for  this 
reduction  in  SNR.  The  first  is  due  to  the  data  acquisition  parameter  constraints  imposed  by  the 
dynamic  survey  mode  (i.e.,  reduced  stacking,  positioning  ambiguity,  etc.).  These  constraints  are 
fundamental  to  dynamic  surveys  and  will  need  to  be  overcome  through  optimal  selection  of  data 
acquisition  parameters  and  data  inversion  methodologies  as  described  in  the  previous  section. 

The  second  cause  of  reduced  SNR  is  due  to  hardware  constraints  associated  with  the  current 
sensor  design.  Because  the  OPTEMA  uses  the  MetalMapper  data  acquisition  architecture,  the 
system  is  required  to  operate  within  the  limits  imposed  by  these  electronics.  These  limits  include 
most  notably  the  transmitter  current  and  kickback  voltage  limits.  The  OPTEMA  provides  an 
almost  effective  doubling  in  coverage  area  compared  to  that  provided  by  the  MetalMapper  (1.8 
meter  swath  vs.  1.0  meter  swath).  This  larger  swath  is  produced  by  a  larger  effective  total 
transmitter  area;  however,  due  to  the  kickback  voltage  limits  associated  with  the  MetalMapper 
electronics,  these  transmitters  do  not  carry  the  same  current  levels  as  those  produced  by  the 
MetalMapper  transmitters.  Specifically,  the  OPTEMA  transmitters  operate  at  approximately  2/3 
the  current  level  of  the  MetalMapper  transmitters.  This  reduced  current  translates  to  sub-optimal 
data  SNR.  While  the  results  of  our  APG  demonstration  indicate  excellent  detection 
performance,  we  believe  data  SNR  could  be  further  improved  prior  to  any  future  surveys  through 
a  modification  to  the  transmitter  driver  components. 
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3.0  PERFORMANCE  OBJECTIVES 

Performance  objectives  for  the  proposed  demonstration  are  summarized  in  Table  3. 


Table  3.  Performance  Objectives. 


Performance 

Objective 

Metric 

Success 

Criteria 

Results  Achieved 

Quantitative  Performance  Objectives 

Blind  Grid 

Open  Field 

Detection  of 
all  TOIs 

Probability  of 
detection  (P^) 
of  all  seeded 
TOIs 

•  Dynamic 
OPTEMA 
data 

•  Ranked 
anomaly  list 

•  Aberdeen 
Test  Center 
(ATC) 
scoring 
report 

Prf  >=0.98  for 
all  TOIs 
encountered 

Yes  P^l.O 
(100%  TOI 
identified) 

Yes  P^O.99 
(99%  TOI 
identified) 

Maximize 

False 

Positive 
Rejection 
(FPR)  Rate  at 
maximum  P^ 
value 

Number  of 
non-TOIs 
correctly 
classified  as 
non-TOIs  out 
of  total 
number  of 
non-TOIs 

•  Dynamic 
OPTEMA 
data 

•  Ranked 
anomaly  list 

•  ATC 
scoring 
report 

EPR>=0.60 
for  all  TOIs 
detected  (i.e., 
at  least  60% 
digs  saved 
with 

maximum 
detection  of 
TOIs) 

Yes 

EPR=0.85 
(85%  clutter 
rejection) 

Yes 

EPR=0.83 
(83%  clutter 
rejection) 

Maximize 

correct 
classification 
of  TOIs 

Number  of 

TOIs  classified 
as  correct  type 
out  of  total 
number  of 

TOIs 

encountered 

•  Dynamic 
OPTEMA 
data 

•  Ranked 
anomaly  list 

•  ATC 
scoring 
report 

Pcc>=0.90 
(i.e.,  90%  or 
greater 
correct 

classification) 

Yes  Vcc  =0.93 
(93%  of  TOI 
correctly 
classified 
according  to 
type) 

Yes  Pec  =0.90 
(90%  of  TOI 
correctly 
classified 
according  to 
type) 

Accurate 
estimation  of 
target 
locations 

Northing, 
Easting,  and 
depth  mean 
location  error 
(MLE) 

•  Dynamic 
OPTEMA 
data 

•  Estimated 
target 
locations 

•  ATC 
scoring 
report 

N,  E,  depth 
MLE  <=.10m 

Yes 

Depth 
MLE=.032 
(Northing  and 
Easting  MLE 
N/A  for  grid) 

Yes 

Depth 

MLE=.01; 

Northing 

MLE=.01; 

Easting 

MLE=-.02 
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Precise 
estimation  of 
target 
locations 

Northing, 
Easting,  and 
depth  error 
standard 
deviation 
(SDV) 

•  Dynamic 
OPTEMA 
data 

•  Estimated 
target 
locations 

•  ATC 
scoring 
report 

N,  E,  depth 
SDV  <=.10m 

Yes 

Depth 
SDV=.053 
(Northing  and 
Easting  SDV 
N/A  for  grid) 

Yes 

Depth=.07; 

Northing=.06; 

Easting=-.07 

Production 

rate 

Effective  area 
surveyed  per 
hour  of 
operation 

•  Eield  logs, 
data  file 
time  stamps 

>=0.5  acre/hr 

Yes 

0.50acre/hr 

No 

0.30  acre/hr 

Qualitative  Performance  Objectives 

Ease  of  use 

•  Operator 
feedback 
regarding 
ease  of 
operation 

(e-g-, 

navigation, 

speed 

control, 

etc.) 

Yes 

Yes 

3.1  OBJECTIVE:  DETECTION  OF  ALL  TARGETS  OF  INTEREST 

Our  objective  is  to  detect  all  seeded  TOIs  encountered  for  data  collected  in  the  blind  grid  and 
indirect  fire  areas.  This  test  establishes  the  OPTEMA’s  ability  to  function  as  a  detection  sensor. 

3.1.1  Data  Requirements 

Detections  were  identified  from  the  OPTEMA  dynamic  survey  data  acquired  in  the 
aforementioned  areas.  We  applied  a  threshold  based  on  noise  floor  standard  deviation  to  identify 
ROIs  for  further  classification.  Data  collected  in  the  calibration  grid  were  used  to  establish  the 
optimal  threshold  and  subset  of  data  channels  for  detection  (e.g.,  Transmit  Z  -  Receive  Z, 
Transmit  Y  -  Receive  Y,  etc.).  We  used  ground  truth  data  for  the  calibration  grid  to  adjust 
detection  parameters.  Analysis  of  blind  data  sets  required  ATC  scoring  to  determine  detection 
performance.  A  detection  was  considered  valid  if  the  ground  truth  TOI  position  fell  within  a 
0.5m  radius  of  the  detection  coordinates. 

3.1.2  Metric 

This  objective  applies  a  Probability  of  Detection  (P^)  metric  to  assess  performance.  Pj  values  for 
each  area  were  generated  by  taking  the  ratio  of  the  total  number  of  TOIs  detected  to  the  total 
number  of  TOIs  encountered  by  the  OPTEMA. 

3.1.3  Success  Criteria 

A  Prf  >=0.98  is  considered  successful. 
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3.1,4  Results  Achieved 

We  achieved  a  =1.0  (100%  TOls  detected)  in  the  blind  grid  and  a  =0.99  (99%  TOls 
detected)  in  the  indirect  fire  area.  The  objective  was  met  for  both  areas. 

3.2  OBJECTIVE:  MAXIMIZE  FALSE  POSITIVE  REJECTION  RATE  (FPR) 

Our  objective  is  to  maximize  rejection  of  false  positives  (i.e.,  unnecessary  digs)  while 
maintaining  correct  identification  of  all  TOls  detected  (i.e.,  with  Pj  value  maximized). 

3.2.1  Data  Requirements 

We  applied  classification  to  all  ROls  generated  from  the  detections  identified  in  the  dynamic 
survey  data.  Each  ROl  contained  a  subset  of  soundings  from  the  dynamic  data  that  corresponded 
to  the  anomaly  source.  Based  on  the  results  of  inverting  these  data,  we  generated  a  ranked  list  of 
likely  TOls.  We  placed  highest  confidence  TOls  at  the  top  of  the  list  and  highest  confidence 
non-TOls  (i.e.,  clutter)  at  the  bottom  of  the  list.  Our  objective  was  to  rank  the  list  such  that  the 
last  TOl  identified  on  the  list  would  be  ranked  above  at  least  60%  of  the  non-TOls.  This  list  was 
submitted  to  ATC  for  ground  truth  scoring. 

3.2.2  Metric 

This  objective  applies  a  False  Positive  Rejection  (FPR)  rate.  FPR  values  for  each  area  were 
generated  by  taking  the  ratio  of  the  number  of  non-TOls  ranked  below  the  last  TOI  on  the  list  out 
of  the  total  number  of  non-TOls  on  the  list. 

3.2.3  Success  Criteria 

An  FPR  value  >=0.60  (60%  clutter  rejection)  is  considered  successful. 

3.2.4  Results  Achieved 

We  achieved  an  FPR=0.85  (85%  clutter  rejection)  in  the  blind  grid  and  an  FPR=0.83  (83% 
clutter  rejection)  in  the  indirect  fire  area.  The  objective  was  met  for  both  areas. 

3.3  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOIS 

Our  objective  is  to  correctly  classify  TOls  by  type. 

3.3.1  Data  Requirements 

We  applied  classification  to  all  ROls  generated  from  the  detections  identified  in  the  dynamic 
survey  data.  Each  ROl  contained  a  subset  of  soundings  from  the  dynamic  data  that  corresponded 
to  an  anomaly  source.  Based  on  the  results  of  inverting  these  data,  we  generated  polarizability 
curves  corresponding  to  each  anomaly  source.  We  compared  these  polarizabilities  to  those  of 
known  TOls  in  each  area  to  classify  each  ranked  TOl  by  type.  These  classifications  were 
recorded  on  the  ranked  anomaly  list  submitted  to  ATC  for  ground  truth  scoring. 

3.3.2  Metric 

This  objective  applies  a  Probability  of  Correct  Classification  (Pec)  metric.  Pec  values  were 
generated  for  each  area  by  taking  the  ratio  of  the  number  of  TOls  correctly  classified  by  type  out 
of  the  total  number  of  TOls  encountered. 

3.3.3  Success  Criteria 

A  Pcc>=0.90  (90%  correct  classification)  is  considered  successful. 

3.3.4  Results  Achieved 
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We  achieved  a  Pcc=0.93  (93%  correct  classification)  in  the  blind  grid  and  a  Pcc=0.90  (90% 
correct  classification)  in  the  indirect  fire  area.  The  objective  was  met  for  both  areas. 

3.4  OBJECTIVE:  ACCURATE  AND  PRECISE  TARGET  LOCATION  ESTIMATION 

Our  objective  is  to  provide  accurate  and  precise  estimates  of  TOI  x,  y,  and  z  locations  (i.e., 
Easting,  Northing,  and  depth  positions). 

3.4.1  Data  Requirements 

We  applied  classification  to  all  ROIs  generated  from  the  detections  identified  in  the  dynamic 
survey  data.  Each  ROI  contained  a  subset  of  soundings  from  the  dynamic  data  that  corresponded 
to  an  anomaly  source.  Based  on  the  results  of  inverting  these  data,  we  extracted  position  and 
orientation  estimates  for  each  TOI.  These  position  and  orientation  estimates  were  recorded  on 
the  ranked  anomaly  list  submitted  to  ATC  for  ground  truth  scoring. 

3.4.2  Metric 

Mean  Eocation  Error  (MLE)  and  location  error  Standard  Deviation  (SDV)  were  used  as  metrics 
to  gauge  TOI  localization  accuracy  and  precision.  The  MLE  value  was  calculated  by  taking  the 
mean  error  value  of  the  Northing,  Easting,  and  depth  location  estimates  for  all  TOIs.  The  SDV 
value  was  calculated  by  taking  the  standard  deviation  for  each  of  these  errors. 

3.4.3  Success  Criteria 

An  MLE<=.10m  and  SDV<=.10m  is  considered  successful. 

3.4.4  Results  Achieved 

We  achieved  a  depth  MLE=.032m  with  SDV=.053m  in  the  blind  grid  (Northing  and  Easting 
metrics  are  not  applicable  for  the  grid  area).  We  achieved  a  depth  MLE=.01m  with  SDV=.07m, 
an  Easting  MLE=-.02m  with  SDV=.06m,  and  a  Northing  MLE=.01m  with  SDV=.07m.  The 
objective  was  met  for  both  areas. 

3.5  OBJECTIVE:  PRODUCTION  RATE 

Production  rate  is  a  useful  indicator  of  the  overall  efficiency  of  the  one-pass  method.  It  provides 
a  quantifiable  metric  to  be  used  for  comparison  to  combined  production  rates  associated  with 
DGM  surveys  and  cued  surveys. 

3.5.1  Data  Requirements 

To  determine  production  rate,  we  used  the  detection  maps  to  identify  the  total  survey  area 
covered  by  the  OPTEMA  system.  We  then  consulted  field  logs  and  file  time  stamps  to 
determine  an  effective  survey  time.  Production  time  included  time  required  to  collect  data  along 
each  transect,  time  required  to  maneuver  the  vehicle  around  obstacles  and  turn  the  vehicle 
around  at  the  end  of  each  line,  and  time  required  for  regular  equipment  checks  (e.g.,  battery 
voltage  check,  operation  parameter  adjustments,  etc.).  To  the  extent  possible,  we  did  not  include 
time  spent  on  inadvertent  system  down-time  associated  with  delays  not  related  to  standard 
production  operations;  however,  it  proved  challenging  to  isolate  these  instances  in  the  indirect 
fire  survey  data.  Excessive  amounts  of  standing  water  in  the  indirect  fire  area  required  careful 
repositioning  of  the  array  around  deep  water  holes.  In  several  instances,  the  vehicle  got  stuck  in 
deep  holes  and  required  several  minutes  to  rectify.  As  a  result,  data  collection  along  almost 
every  transect  in  the  indirect  fire  area  was  affected  by  unusually  challenging  conditions.  This  led 
to  a  lower  overall  production  rate  for  the  indirect  fire  area. 
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3.5.2  Metric 

Production  rate  was  calculated  based  on  the  average  area  surveyed  per  hour. 

3.5.3  Success  Criteria 

A  production  rate  of  0.5acre/hr  or  greater  is  considered  successful. 

3.5.4  Results  Achieved 

We  achieved  a  production  rate  of  0.50acre/hr  in  the  blind  grid  and  a  production  rate  of 
0.30acre/hr  in  the  indirect  fire  area.  The  objective  was  met  for  the  blind  grid,  but  was  not  met  for 
the  indirect  fire  area.  The  aforementioned  excessive  levels  of  standing  water  in  the  indirect  fire 
area  led  to  the  lower  than  expected  production  rate  for  this  area. 


White  River  Technologies 
APG  Demonstration  Report 


18 


September  2014 


4.0  SITE  DESCRIPTION 


This  demonstration  was  conducted  at  the  Standardized  UXO  Technology  Demonstration  Site 
located  at  the  Aberdeen  Proving  Ground  in  Aberdeen,  MD.  The  site  comprises  several  test  areas, 
most  of  which  are  located  in  open,  relatively  flat,  grassy  areas.  The  open  field  has  several  low- 
lying  areas,  which  tend  to  accumulate  high  levels  of  standing  water  during  certain  times  of  year. 

4.1  SITE  SELECTION 

This  site  was  selected  for  our  demonstration  since  it  serves  as  one  of  the  two  principal  proving 
grounds  (along  with  YPG)  for  emerging  munitions  classifications  sensors.  Consequently,  results 
generated  from  data  collected  at  this  site  provide  good  quantification  of  the  relative 
performances  of  various  sensor  technologies. 

Because  OPTEMA  is  a  large  towed  array  designed  for  maximum  efficiency  in  open  field  areas, 
we  selected  areas  of  the  site  that  were  amenable  to  towed  array  operations.  We  collected  data  in 
three  areas  at  the  UXO  Test  Site:  1)  the  calibration  grid;  2)  the  blind  grid;  and  3)  the  indirect  fire 
area.  These  areas  provided  a  good  sample  of  target  types  and  emplacement  depths  for  gauging 
sensor  detection  and  classification  performance.  Figure  1 1  provides  a  diagram  that  depicts  the 
site  layout. 


Figure  11.  Diagram  of  APG  UXO  Technology  Demonstration  Site  showing  areas  surveyed.  The  OPTEMA 
demonstration  was  conducted  in  the  calihration  grid,  blind  grid,  and  indirect  fire  areas. 
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During  our  two  weeks  at  the  site,  we  experieneed  several  significant  rain  events.  Those  events, 
combined  with  the  recent  spring  thaw  meant  that  water  levels  were  very  high,  particularly  in 
several  portions  of  the  indirect  fire  area  (Figure  12).  Consequently,  there  were  several  large 
exclusion  areas  that  we  were  not  able  to  survey.  Overall,  we  achieved  approximately  70% 
coverage  of  the  indirect  fire  area. 


Figure  12.  OPTEMA  survey  iu  the  iudirect  fire  area.  Excessive  levels  of  staudiug  water  due  to  receut  raius 
aud  spriug  thaw  produced  several  exclusiou  areas.  Overall,  approximately  70%  coverage  was  achieved  iu  the 
iudirect  fire  area. 
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5.0  TEST  DESIGN 


The  field  eomponent  of  the  demonstration  was  designed  to  provide  the  data  required  for 
evaluating  the  performanee  objeetives  deseribed  in  seetion  3.  Details  of  the  field  test  are 
deseribed  in  the  following  subseetions. 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  field  test  eomprised  dynamie  OPTEMA  surveys  eondueted  in  the  ealibration  grid,  blind  grid, 
and  indireet  fire  areas  of  the  APG  UXO  Teehnology  Demonstration  Site.  Initial  test  aetivities 
were  foeused  on  the  ealibration  grid.  These  initial  test  aetivities  ineluded:  statie  data  eolleetion 
over  ealibration  items  to  generate  polarizability  libraries  for  items  likely  to  be  eneountered  in  the 
blind  areas;  establishment  of  an  Instrument  Verifieation  Strip  (I VS)  from  a  subset  of  targets  in 
the  ealibration  grid;  and  dynamie  surveys  over  the  eomplete  ealibration  grid  using  the  4  sets  of 
data  aequisition  parameters  listed  in  Table  2.  Onee  initial  system  verifieation  was  eompleted,  we 
eondueted  the  remainder  of  the  test  aetivities  in  the  blind  grid  and  indireet  fire  areas  of  the  site. 

We  spent  longer  than  initially  antieipated  in  the  ealibration  area  due  to  faetors  related  to  the 
untested  survey  vehiele.  During  the  period  leading  up  to  the  demonstration,  we  had  eondueted 
tests  at  our  faeility  using  an  available  pickup  truck;  however,  this  vehicle  was  not  acceptable  for 
the  demonstration  due  to  site  restrictions.  Consequently,  we  rented  a  4x4  off-road  utility  vehicle 
that  was  delivered  to  the  site  upon  our  arrival.  The  hitch  on  the  vehicle  was  lower  than 
anticipated  causing  the  leading  edge  of  the  tow  sled  to  drag  on  the  ground.  Initially,  we 
performed  calibration  activities  with  this  configuration;  however,  the  wet  ground  caused  the  sled 
to  dig  in  and  create  added  resistance,  which  proved  too  much  for  the  vehicle’s  traction  and 
resulted  in  several  instances  where  the  vehicle  got  stuck  in  the  mud.  We  resolved  this  problem 
by  modifying  the  boom  on  the  tow  sled  to  lift  the  sled  higher  and  closer  to  the  anticipated 
operational  height.  This  on-site  modification  resolved  the  problem,  but  delayed  blind  area  data 
collection. 

Another  factor  that  contributed  to  the  extension  of  the  calibration  period  was  the  appearance  of 
sporadic  and  very  high  amplitude  noise  spikes  in  the  data.  This  noise  had  never  occurred  during 
any  of  our  pre-demonstration  tests.  Initially,  we  thought  it  might  be  a  failure  of  one  of  the  board 
components  in  the  receiver  electronics;  however,  replacing  with  spares  did  not  resolve  the  issue. 
Ultimately,  the  noise  was  traced  to  an  inverter  used  to  power  the  data  acquisition  laptop. 
Although,  the  laptop  had  no  power  connections  to  the  OPTEMA  DAQ,  the  Ethernet  cable  was 
enough  to  couple  the  noise.  We  resolved  the  issue  by  replacing  the  laptop  batteries  with  spares 
throughout  the  day  instead  of  utilizing  vehicle-supplied  power  for  the  inverter  and  laptop. 

Once  we  resolved  the  noise  and  tow  issues,  we  continued  test  activities  in  the  blind  areas.  The 
remaining  blind  test  surveys  were  conducted  efficiently  with  the  exception  of  some  delays  in  the 
indirect  fire  area  caused  by  excessive  levels  of  standing  water.  Our  tow  vehicle  got  stuck  on 
several  occasions  and  we  frequently  had  to  reposition  the  array  around  deep  water  holes.  Table  4 
provides  a  summary  of  our  field  activity  schedule. 
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Table  4.  Schedule  of  APG  field  activities. 


Site  Restrictions 


Demobilization 


□ 


5.2  SITE  PREPARATION 

Because  this  demonstration  was  conducted  at  one  of  the  Standardized  UXO  Technology 
Demonstration  Sites,  no  special  preparation  was  required. 

5.3  SYSTEM  SPECIFICATION 

The  OPTEMA  comprises  five  transmitter  coils  and  fourteen  3-axis  receiver  cubes.  The  five 
transmitters  include  one  large  base  Z-directed  (vertical  axis)  transmitter  that  surrounds  two  pairs 
of  X’-  and  Y’-  directed  (horizontal  axis)  transmitters.  The  Z-transmitter  dimensions  are  1.8m  x 
1.0m  and  the  X’-  and  Y’-  (primed  coordinates  denote  45  degree  rotation  to  primary  reference 
frame)  transmitters  are  1.22m  x  1.0m.  Figure  13  shows  a  diagram  of  the  transmitter  layout  and 
dimensions. 


Figure  13.  OPTEMA  trausmitter  layout  aud  dimeusious. 

The  receivers  are  the  standard  G&G  Sciences  10cm  cubes  developed  for  advanced  EMI  sensors 
(Figure  14).  These  cubes  are  rotated  45  degrees  to  the  main  reference  frame,  in-line  with  the 
horizontal  axis  transmitters.  Figure  14  shows  the  layout  of  the  receiver  cubes  within  the 
OPTEMA  sensor  head. 
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Rx  positions:  [x  y  z]  (unprimed): 

rxl=[-.765  .107  0.01] ; 
rx2=[-.765  -.162  0.01] ; 
rx3=[-.629  -.032  0.01] ; 
rx4=[-.412  .242  0.01] ; 
rx5=[-.413  -.257  0.01] ; 
rx6=[-.188  -.046  0.01] ; 
rx7=[ .042  .182  0.01] ; 
rx8=[-.026  -.205  0.01] ; 
rx9=[ .200  .017  0.01] ; 
rxl0=[ .419  .237  0.01] ; 
rxll=[ .422  -.241  0.01] ; 
rxl2=[.626  .044  0.01] ; 
rxl3=[ .761  .173  0.01] ; 
rxl4=[ .770  -.107  0.01] ; 

Figure  14.  OPTEMA  sensor  head  receiver  cube  layout.  Receiver  cubes  are  10cm. 

The  OPTEMA  sensor  head  is  mounted  in  a  non-metallic  tow  sled.  The  sled  features  a  wooden 
tow  boom,  a  protective  skid  plate,  and  low  pressure  pneumatic  tires.  The  tires  provide  shock 
absorption,  reduce  wear  on  the  sled  frame,  and  provide  lower  tow  resistance.  In  flat  terrain,  the 
sled  rolls  on  the  aft-mounted  wheels;  however,  the  skid  plate  provides  additional  support  in  more 
challenging  terrain  conditions.  The  tow  bar  puts  the  leading  edge  of  the  OPTEMA  sensor  head 
approximately  three  meters  behind  the  hitch  point  on  the  vehicle.  Figure  15  shows  a  picture  of 
the  sensor  head  mounted  in  the  tow  sled. 
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Figure  15.  OPTEMA  tow  sled. 


The  OPTEMA  incorporates  a  Trimble  Real  Time  Kinematic  Differential  GPS  for  sensor  head 
position  data.  A  Trimble  R8  receiver  is  mounted  directly  above  the  center  of  the  sensor  head. 
Inertial  measurements  are  provided  by  a  Microstrain  3DM-GX3-25  orientation  sensor.  The 
Euler  angle  outputs  provide  pitch,  roll,  and  yaw  measurements  for  the  OPTEMA  sensor  head. 
The  IMU  is  co-located  with  the  GPS  receiver. 

OPTEMA  sensor  electronics  include  the  transmitter  and  receiver  modules,  data  acquisition 
computer,  and  power  inverter.  Electronics  are  mounted  in  a  2’x  2’x  3’  (LxWxH)  enclosure  that 
can  be  fastened  to  the  bed  of  any  tow  vehicle.  A  pair  of  deep  cycle  marine  batteries  provide  +/- 
12V  for  the  transmitter  and  receiver  modules.  A  single  deep  cycle  battery  provides  12V  for  the 
inverter,  which  supplies  power  for  the  data  acquisition  computer.  An  operator  PC  can  be 
connected  to  the  data  acquisition  computer  via  remote  desktop  through  an  Ethernet  cable. 

The  transmitter  controller  provides  switching  at  60  Hz  subharmonics  at  50%  duty  cycle.  This 
design  produces  a  bipolar  square  waveform  with  a  user  selected  period.  The  waveform  period 
determines  the  decay  window  measured  by  the  receivers.  Typically,  2.8ms  or  8.3ms  decay 
windows  are  used  for  dynamic  survey  mode  (see  Table  2  for  reference).  The  number  of 
waveform  repeats  in  each  data  block  (a  single  Transmit-Z,  Transmit-Y,  or  Transmit-X  sequence) 
is  a  function  of  the  waveform  period  selected.  Eigure  16  shows  an  example  of  a  sequence  of  3 
data  blocks  corresponding  to  an  8.3ms  decay  period. 
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Figure  16.  Waveform  corresponding  to  3  OPTEMA  data  blocks.  Each  block  comprises  one  sequence  of 
Transmit-Z,  Transmit-Y,  or  Transmit-X.  The  number  of  base  waveform  repeats  for  each  transmitter  cycle 
depends  on  the  decay  period  selected.  This  example  shows  an  8.3ms  decay  period.  This  period  produces  only 
1  repeat  for  each  transmitter  cycle.  If  stacking  is  increased,  this  complete  sequence  (three  blocks)  is  repeated 
and  the  result  is  averaged. 

For  the  demonstration  we  used  the  data  acquisition  parameters  shown  in  Table  2. 

5.4  CALIBRATION  ACTIVITIES 

Calibration  activities  at  the  demonstration  site  consisted  of  both  initial  and  daily  calibration 
routines.  We  performed  initial  system  verification  and  baseline  measurements  in  the  calibration 
grid.  Initial  calibration  activities  included  basic  instrument  verification  measurements  such  as 
noise  and  static  (spike)  tests.  These  instrument  verification  measurements  were  conducted  with 
the  OPTEMA  system  located  in  a  clean  (anomaly  free)  area  near  the  calibration  grid. 

The  noise  test  established  a  baseline  noise  floor  for  the  OPTEMA  system.  Eor  the  noise  test,  the 
system  was  stationed  over  the  cleared  location  and  set  to  run  in  a  continuous  (dynamic)  mode 
where  each  measurement  data  block  was  recorded  as  a  separate  data  point  (i.e.,  stacking  set  to  1). 
Typically,  N=30  or  more  data-points  were  collected.  To  calculate  the  OPTEMA  noise  floor,  the 
standard  deviation  of  the  N  data  points  of  each  time-channel  for  each  of  the  126  transmitter- 
receiver  combinations  was  calculated.  This  process  helps  to  identify  any  significant  deviations  in 
sensor  noise  on  a  day-to-day  basis.  Significant  deviations  in  data  channel  noise  may  be 
indicative  of  a  hardware  fault.  An  example  of  instrument  noise  standard  deviation  is  shown  in 
Eigure  17.  This  test  was  performed  on  an  initial  basis  for  baseline  reference,  and  subsequently 
on  a  daily  basis  for  verification.  During  our  blind  area  test  activities,  no  significant  deviations  in 
sensor  noise  were  noted. 
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Figure  17.  Example  of  instrument  noise  standard  deviation  for  seven  receiver  cubes.  The  red  line  represents 
a  baseline  quality  objective  threshold.  The  magenta,  cyan  and  blue  lines  correspond  to  the  z-,  y-,  and  x- 
component  receiver  data. 

Static,  or  spike,  tests  were  used  to  verify  consistency  in  data  channel  output  on  a  daily  basis. 
Static  tests  were  performed  with  the  OPTEMA  stationed  in  a  clean  area  near  the  calibration  grid. 
After  setting  the  OPTEMA  data  acquisition  parameters  to  static  mode  (Table  1),  a  calibration 
ball  was  placed  in  a  test  jig  centered  over  each  receiver  cube  (Eigure  18).  This  test  provided 
another  measurement  of  instrument  consistency.  The  calibration  ball  response  in  each  principal 
data  channel  (i.e.,  Transmit-Z/Receive-Z,  Transmit- Y/Receive-Y,  Transmit-X/Receive-X) 
should  be  repeatable  within  10%  deviation  on  each  test  (some  deviation  may  be  caused  by  small 
jig  placement  inconsistencies).  Any  significant  deviations  in  the  calibration  ball  response  may 
be  indicative  of  hardware  faults.  An  example  of  the  principal  data  channel  calibration  ball 
response  is  shown  in  Eigure  19.  This  test  was  performed  on  an  initial  basis  for  baseline 
reference,  and  subsequently  on  a  daily  basis  for  verification. 

During  our  blind  test  activities,  only  one  instance  occurred  where  a  significant  deviation  in  spike 
amplitude  was  noted.  This  deviation  occurred  as  a  result  of  a  broken  receiver  channel  connector 
in  the  DAQ.  The  strain  relief  for  our  cables  had  slipped  causing  one  of  the  receiver  cables  to  pull 


White  River  Technologies 
APG  Demonstration  Report 


26 


September  2014 


the  connector  and  break  one  of  the  solder  joints.  This  malfunction  affected  7  data  files  in  the 
indirect  fire  area,  which  were  subsequently  recollected. 


Figure  18.  Calibration  ball  spike  test.  A  calibration  ball  is  placed  over  each  receiver  cube  in  a  repeatable 
location  to  identify  any  inconsistencies  in  data  cbannel  output. 


Id''  10°  10"''  10°  lo’’'  10° 

Time  (ms)  Time  (ms)  Time  (ms) 

Figure  19.  Calibration  ball  response  in  principal  Tx-Rx  pairings  for  seven  receiver  cubes.  Tbe  red  line 
indicates  tbe  reference  measurement;  tbe  blue  line  indicates  tbe  measured  response. 
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Initial  calibration  activities  also  included  statie  measurements  over  calibration  grid  items.  These 
static  data  were  used  to  generate  polarizability  libraries  for  TOIs  in  the  blind  grid  and  indirect 
fire  areas.  For  each  TOI  type  in  the  calibration  grid,  we  acquired  3-4  static  measurements  with 
eaeh  X’-Af’-  pair  of  transmitters  approximately  centered  over  the  target.  By  eentering  each 
transmitter  pair  over  the  target,  we  were  able  to  measure  the  response  aeross  the  complete  sensor 
swath;  ensuring  consistent  results  were  achieved  using  any  subset  of  data  channels  within  the 
array. 

Referenee  calibration  grid  polarizabilities  were  also  used  on  a  daily  basis  for  instrument 
verification.  We  collected  static  data  over  a  subset  of  targets  in  the  calibration  grid  (3-4  targets) 
regularly  to  ensure  that  recovered  polarizabilities  were  consistent  on  a  day-to-day  basis.  These 
targets  served  as  an  I  VS.  Polarizabilities  reeovered  from  I  VS  data  should  mateh  the  reference 
libraries  with  95%  fit  for  static  measurements  collected  with  each  X’-Af’-  transmitter  pair 
centered  over  the  target.  Additionally,  estimated  target  location  coordinates  recovered  from 
inversion  of  the  IVS  data  should  be  within  15em  of  the  ground  truth  eoordinates.  Thus,  this  test 
served  as  a  measurement  of  instrument  eonsisteney  as  well  as  model  eonsistency.  Figure  20 
shows  an  example  of  IVS  data  matched  to  library  polarizabilities. 


Figure  20.  Polarizabilities  recovered  from  daily  IVS  data  matched  to  a  library  reference.  Red,  black  and  blue 
lines:  primary,  secondary  and  tertiary  recovered  polarizabilities.  Grey  lines:  reference  polarizabilities.  IVS 
items  shown  here  are  medium  ISOs.  All  fits  are  within  95%  match. 

Instrument  verification  measurements  were  eonducted  on  a  twiee  daily  (morning  and  afternoon) 
basis  to  ensure  consistency  throughout  each  day  of  operation.  During  our  blind  test  activities,  all 
objeetives  were  met  for  the  loeation  and  fit  aeeuraey  metrics. 
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5.5  DATA  COLLECTION  PROCEDURES 


Throughout  the  demonstration,  we  performed  OPTEMA  data  collection  procedures  in  both  static 
and  dynamic  modes.  The  static  data  collection  mode  was  used  for  initial  library  data  collection 
and  daily  instrument  verification  tests.  Dynamic  mode  was  used  for  surveys  conducted  in  the 
calibration  grid,  blind  grid,  and  indirect  fire  areas.  Table  1  and  Table  2  (section  2.1)  show  the 
default  static  and  dynamic  mode  acquisition  parameter  settings,  respectively. 

We  performed  static  measurements  only  in  the  calibration  grid.  These  measurements  required  an 
initial  background  measurement  acquired  in  a  clean  area  (anomaly  free)  of  the  calibration  grid. 
This  measurement  was  then  used  for  background  subtraction  from  subsequent  static 
measurements  made  in  the  grid.  We  used  static  measurements  for  library  data  collection,  I  VS 
measurements,  and  daily  spike  tests.  Static  measurements  over  targets  were  made  with  both 
pairs  of  horizontal  axis  transmitters  centered  over  the  target  to  verily  measurement  consistency 
across  the  entire  swath  of  the  array. 

We  performed  dynamic  measurements  as  part  of  calibration  grid,  blind  grid,  and  indirect  fire 
area  surveys.  We  conducted  an  initial  dynamic  survey  in  the  calibration  grid.  This  survey 
comprised  full  sensor  coverage  of  the  grid  four  times  using  each  of  the  four  default  settings  listed 
in  Table  2.  Figure  21  shows  an  anomaly  map  generated  from  full  coverage  OPTEMA  data 
collected  in  the  calibration  grid. 


^^'RT  Data  CoUvction  Ditpbv  Tool 


Figure  21.  OPTEMA  data  map  (400  -  500  microsecond  time  gate  sum  of  Transmit  Z-/Receive  Z-  channels)  of 
the  calihration  grid. 

After  collecting  these  calibration  grid  data,  we  performed  an  in-field  analysis  of  the  data 
corresponding  to  each  parameter  setting  to  down-select  two  optimal  settings  for  the  site.  This 
decision  was  based  on  library  fits  for  the  polarizabilities  recovered  from  each  dynamic  data  set. 
We  selected  settings  1  and  3  in  Table  2  (nStk  =  1  for  the  8.3ms  and  2.8ms  decays,  respectively). 
These  settings  provided  a  down  track  sample  resolution  of  approximately  5  cm.  Our  intent  was 
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for  the  2.8ms  decay  to  provide  slightly  better  SNR  due  to  the  increase  in  the  number  of  repeats  in 
the  data  block.  We  performed  dynamic  surveys  in  the  blind  grid  using  each  of  the  two  down- 
selected  parameter  settings;  however,  further  analysis  indicated  that  the  8.3ms  decay  setting 
provided  better  information  for  classification  (even  at  the  slightly  elevated  noise  levels). 
Therefore,  we  selected  only  the  8.3ms  setting  (setting  1  in  Table  2)  for  the  open  field  survey. 

Background  measurements  were  not  required  for  dynamic  data  collection.  Because  dynamic 
surveys  cover  both  anomalies  and  background  regions,  background  measurements  can  be  pulled 
directly  from  the  dynamic  data  to  be  used  for  subtraction  from  the  ROIs. 

Navigation  for  both  static  and  dynamic  measurements  was  provided  by  the  EM3DAcquire 
navigation  interface.  We  generated  line  segment  files  (for  dynamic  measurements)  and  cued 
files  (for  static  measurements)  prior  to  conducting  surveys  in  the  UXO  Test  Site  areas.  Ground 
truth  for  the  calibration  grid  was  used  to  generate  cues  for  each  of  the  items  in  the  calibration 
grid.  Grid  coordinates  for  all  three  areas  were  required  to  generate  transect  lines  in  each  survey 
area.  The  format  for  each  cue  is  a  standard  longitude,  latitude  coordinate  in  decimal  degrees. 
The  format  for  line  segments  is  the  longitude,  latitude  of  each  line  endpoint  in  decimal  degrees. 
We  used  a  1.2m  spacing  for  transect  lines  in  dynamic  survey  areas.  This  provided  a  33%  overlap 
for  adjacent  transects. 

Dynamic  data  files  can  be  quite  large.  Depending  on  the  acquisition  parameters  selected,  the 
OPTEMA  generates  data  samples  at  a  rate  of  5-10  Hz.  This  can  produce  as  much  as  250MB  for 
10  minutes  of  operation.  To  accommodate  the  large  volume  of  data,  we  used  an  external  drive  to 
store  data  and  periodically  transfer  data  from  the  data  acquisition  hard  drive  to  ensure  that 
sufficient  memory  was  available  for  the  duration  of  dynamic  survey  operations. 

We  performed  quality  checks  of  all  data  collected  on  a  daily  basis.  Specifically,  we  looked  for 
discrepancies  in  receiver  channel  output  that  would  indicate  sensor  faults.  In  addition  to  the 
aforementioned  instrument  verification  activities,  we  also  performed  visual  quality  control  for  all 
dynamic  data  by  spot-checking  samples  along  each  data  transect.  Our  daily  quality  checks 
identified  two  instances  of  incomplete  line  collections  in  the  blind  grid  caused  by  operator  error. 
These  lines  were  subsequently  recollected. 

5.  6  VALIDATION 

Upon  completion  of  the  UXO  Technology  Demonstration  Site  data  collection,  we  generated 
ranked  anomaly  lists  using  the  standard  ATC  scoring  formats.  Ranked  lists  for  each  scoring  area 
included  confidence  rankings  of  anomalies  in  the  blind  areas  with  highest  confidence  TOI 
rankings  at  the  top  of  the  list  and  highest  confidence  non-TOI  rankings  at  the  bottom  of  the  list. 
The  ranked  lists  also  included  the  estimated  location,  depth,  and  orientation  for  anomalies 
identified  as  TOIs  and  the  estimated  location  for  anomalies  identified  as  clutter.  These  lists  were 
submitted  to  ATC  for  scoring. 

To  ensure  results  for  the  blind  data  analysis  were  not  influenced  by  prior  knowledge  of  the 
survey  area;  analysts  who  had  never  performed  any  previous  analysis  of  APG  blind  areas 
conducted  the  analysis  of  the  OPTEMA  data  set. 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 


OPTEMA  data  analysis  includes  preprocessing,  detection  processing,  and  classification 
processing  and  analysis  stages. 

6.1  PREPROCESSING 

Preprocessing  steps  are  similar  for  both  static  and  dynamic  data.  A  transmitter  current 
normalization  is  performed  on  all  OPTEMA  data  files  by  dividing  all  data  channel  values  by  the 
peak  current  value  corresponding  to  the  appropriate  transmitter  (e.g..  Transmit- Y/Receive-Z  data 
channels  are  divided  by  the  peak  Y-transmitter  current).  This  process  ensures  that  only  the  size 
and  number  of  windings  (both  of  which  are  constant)  for  each  transmitter  affect  the  response 
measured.  Thus,  no  discrepancies  in  transmitter  current  (which  may  vary)  will  infiuence  the 
data.  Current  normalization  is  performed  once  for  a  static  data  file  and  is  repeated  for  each 
sounding  in  a  dynamic  file. 

After  each  data  file  is  current-normalized,  a  background  subtraction  step  is  performed  by 
subtracting  the  value  of  each  data  channel  in  a  background  data  file  from  the  corresponding  data 
channel  value  in  a  data  file.  Background  data  are  collected  with  the  OPTEMA  sensor  stationed 
over  a  clean  (anomaly  free)  area  of  the  site.  Background  subtraction  is  performed  once  for  a 
static  data  file  and  is  repeated  for  each  sounding  in  a  dynamic  file,  for  dynamic  files, 
background  measurements  can  also  be  pulled  directly  from  the  data.  This  process  requires 
identification  of  anomaly  free  locations  in  the  survey  area.  Identification  of  these  regions  is 
performed  during  the  detection  processing  stage. 

6.2  DETECTION  PROCESSING 

Detection  maps  are  created  by  processing  dynamic  data  files.  Detection  processing  requires  data 
gridding  for  each  transmit/receive  pairing.  GPS  latitude/longitude  values  are  converted  to  UTM 
coordinates  (Easting  /  Northing).  Each  sounding  in  a  dynamic  file  is  associated  with  a  GPS 
Easting  and  Northing  coordinate  and  a  measurement  of  sensor  head  pitch,  roll,  and  yaw  angles. 
Receiver  cube  positions  for  each  sounding  are  calculated  by  applying  an  Euler  transformation  to 
the  vector  from  the  center  of  the  GPS  antenna  to  the  center  of  each  receiver  cube.  This 
transformation  creates  an  Easting/Northing  position  for  each  receiver  cube  for  each  sounding. 

for  each  principal  transmit/receive  pairing  (i.e.,  Transmit-Z/Receive-Z,  Transmit- Y/Receive-Y, 
Transmit-X/Receive-X),  a  data  value  is  generated  for  each  receiver  cube  by  summing  the  values 
of  the  middle  time  gates  in  the  corresponding  data  channel.  These  data  values  are  then  mapped 
to  a  2-D  space  using  the  receiver  cube  locations,  finally,  2-D  interpolation  is  applied  to  generate 
the  final  uniformly-spaced  X-,  Y-,  and  Z-  data  maps,  figure  22  shows  the  X-,  Y-,  and  Z-  data 
maps  for  a  37  mm  projectile  buried  at  15  cm  depth. 
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TxY  -  RxY 
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Figure  22.  Example  X-  (left) ,  Y-  (center) ,  and  Z-  (right)  detection  data  maps.  This  example  is  for  a  37  mm 
projectile  hurled  at  15  cm  depth  lying  horizontally  across  the  data  collection  swath. 

The  detection  algorithm  consists  of  two  steps:  1)  apply  a  threshold  based  on  the  data  noise  floor 
standard  deviation  to  create  areas  containing  high  SNR  responses,  and  2)  in  each  high  SNR  area 
apply  a  peak  detection  algorithm  to  create  a  unique  detection  and  ROI  for  each  peak  found.  Step 
2  splits  large  high  response  areas  into  multiple  detections  and  is  an  important  step  in  high  clutter 
areas  or  when  a  small  anomaly  is  in  close  proximity  to  a  large  anomaly.  For  each  detection  a 
radius  is  applied  to  identify  the  ROI  surrounding  each  peak.  The  radius  size  is  based  on  the  local 
gradient  associated  with  the  peak  and  the  number  of  peaks  associated  with  an  anomaly  (1  peak 
for  Z-data;  2  peaks  for  X-  and  Y-data).  Finally,  across  track  and  along  track  indices  are 
generated  for  each  alarm  in  an  ROI.  These  indices  correspond  to  the  receiver  cube  and  sounding 
number  associated  with  each  alarm. 

Each  ROI  is  saved  as  a  data  volume  (number  of  soundings  by  number  of  time  gates  by  number  of 
data  channels)  in  .mat  format.  Alarm  indices  and  UTM  coordinates  are  saved  as  part  of  the  data 
structure  as  well. 

6.3  CLASSIFICATION  PROCESSING 

Following  detection  processing,  all  ROIs  are  further  processed  for  classification  analysis.  The 
first  step  in  classification  processing  is  to  improve  the  relative  point-to-point  positional  accuracy 
within  each  ROI.  Because  the  inversion  algorithm  relies  on  relative,  not  absolute,  positioning 
between  consecutive  soundings  removing  small  absolute  positioning  errors  associated  with  GPS 
accuracy  can  enhance  the  classification  performance.  An  initial  filtering  step  is  performed  by 
comparing  the  calculated  point-to-point  GPS  heading  to  the  value  calculated  by  averaging 
consecutive  IMU  heading  measurements.  The  calculated  GPS  point-to-point  distance  traveled  is 
projected  onto  the  IMU  heading  vector  to  establish  the  relative  position  traveled  between 
consecutive  soundings.  These  relative  positions  are  then  assigned  to  each  sounding  within  the 
detection  ROI. 

Position  vectors  corresponding  to  the  center  of  the  OPTEMA  array  at  each  sounding  location 
within  the  ROI  are  generated  using  the  relative  point-to-point  positions.  These  relative  position 
vectors  are  referenced  to  the  position  of  the  OPTEMA  during  the  peak  response  measurement 
(i.e.,  peak  detection  position  of  the  OPTEMA  center  is  [0  0  0]  in  [x  y  z]). 
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A  second  background  subtraction  is  also  performed.  A  background  measurement  outside  each 
ROI  is  selected  and  subtracted  from  all  soundings  within  the  ROI  to  further  remove  any  localized 
background  response. 

Based  on  the  across  track  and  along  track  indices  for  each  ROI  alarm,  a  subset  of  soundings  is 
selected  from  the  ROI  dynamic  data  for  inversion.  Optimal  soundings  are  those  corresponding 
to  data  collected  while  the  target  was  beneath  one  of  the  horizontal  axis  transmitters  (Figure  23). 
These  data  ensure  the  best  orthogonal  illumination  of  the  target  and  provide  better  constraint  on 
the  inversion. 


Figure  23.  Optimal  target  locations.  Data  are  selected  for  soundings  acquired  while  the  sensor  array  was 
positioned  such  that  the  target  was  located  beneath  one  of  the  highlighted  regions.  These  regions  correspond 
to  areas  of  greater  orthogonaUty  between  transmitter  fields. 

Two  methods  of  inversion  are  applied  to  the  selected  data  within  each  ROI  as  described  in 
section  2.1.  The  first  method  is  to  invert  the  data  in  each  sounding  separately  and  compile  all 
sets  of  polarizabilities  in  a  group  for  further  analysis.  The  second  method  involves  aggregating 
data  from  each  sounding  in  the  subset  and  inverting  the  composite  data.  This  method  applies  the 
relative  position  vectors  to  assign  position  coordinates  to  each  data  point  in  the  aggregated  data 
matrix. 

Finally,  data  are  fit  to  a  dipole-based  forward  model  using  a  least  squares  inversion.  The  forward 
model  accounts  for  the  sensor  array  geometry,  the  position  of  the  array  relative  to  the  target,  and 
the  target  physical  features,  which  are  modeled  as  a  set  of  orthogonal  magnetic  dipoles  (i.e., 
polarizability  tensor).  Once  model  parameters  are  chosen  to  minimize  the  error  between  the  data 
and  the  model  output,  the  resulting  target  polarizabilities  are  selected  for  feature  classification. 

Starting  parameters  for  the  inversion,  such  as  target  position,  are  chosen  based  on  the  detection 
parameters  (i.e.,  across  track  and  along  track  indices  for  each  alarm).  Additionally,  these 
detection  parameters  may  be  used  to  identify  potential  multi-object  scenarios.  For  example,  if  an 
ROI  includes  two  unique  alarms  a  two-source  forward  model  will  be  applied  to  the  data 
inversion. 

6.4  CLASSIFICATION  ANALYSIS 
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For  the  APG  data  set,  we  determined  the  optimal  classification  approach  was  to  invert  each 
sounding  in  the  ROI  on  an  individual  basis.  Using  this  approach,  we  treated  each  sounding  in 
the  ROI  as  an  effective  “cued”  measurement.  Thus,  each  sounding  provided  a  unique 
measurement  at  a  point  along  the  survey  transect.  The  benefit  of  this  approach  was  that  it 
enabled  recovery  of  several  sets  of  polarizabilities  corresponding  to  each  anomaly  source  in  the 
survey  area.  Consequently,  we  were  able  to  improve  confidence  in  our  classification  decision  by 
using  the  statistical  variability  of  the  various  model  parameters  (e.g.,  polarizabilities,  depth, 
location,  orientation,  etc.)  corresponding  to  each  source.  Highly  consistent  model  parameters 
were  a  good  indication  that  the  source  was  well  characterized.  Figure  24  shows  an  example  of 
the  repeatability  of  these  model  parameters  for  a  particular  source. 
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Figure  24.  TOP  LEFT:  OPTEMA  DGM  Map  showing  several  anomalies  in  a  portion  of  the  survey  area.  The 
anomaly  of  interest  is  circled  in  hlack  with  the  model-hased  estimated  location  shown  as  the  magenta  dot. 
TOP  RIGFIT:  Library  matching  for  six  different  UXO  targets.  Soundings  corresponding  to  the  anomaly 
were  selected  for  inversion  and  the  resulting  nine  sets  of  polarizahilities  (each  set  comprises  a  primary, 
secondary,  and  tertiary  polarizahility)  are  plotted  in  magenta  against  the  TOI  library  polarizabilities  (shown 
in  dark  grey).  The  polarizabilities  obtained  from  these  consecutive  soundings  are  almost  identical  and  show  a 
clear  match  to  the  105mm  Indirect  Fire  munition  type.  BOTTOM:  Estimated  model  parameters 
corresponding  to  each  sounding.  Parameters  are  highly  consistent  for  each  sounding,  indicating  a  high 
confidence  decision  can  be  made.  The  Northing  and  Easting  errors  for  each  sounding  are  within  +/-  3cm  of 
the  mean  estimated  location  (shown  as  the  magenta  dot  in  the  DGM  map).  Depth  estimates  are  consistent  to 
within  +/-  2cm  and  orientation  estimates  are  consistent  within  +/-  6  degrees. 
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After  inverting  the  data  eorresponding  to  each  ROI  in  the  blind  survey  areas,  we  analyzed  the 
inverted  model  parameters  to  determine  a  classification  decision  for  each  anomaly  source.  We 
assigned  TOI  (ordnance)  rankings  to  anomalies  based  on  polarizability  fits  to  TOI  libraries.  We 
used  TOI  libraries  for  ordnance  types  known  to  be  located  in  each  area  (25mm,  37mm,  60mm, 
81mm,  105mm,  and  105mm  artillery  for  the  blind  grid;  60mm,  81mm,  and  105mm  for  the 
indirect  fire  area).  Highest  confidence  TOI  rankings  were  assigned  to  anomalies  that  provided 
good  fits  (>90%  match)  for  all  three  polarizabilities.  Lower  confidence  TOI  rankings  were 
assigned  to  anomalies  that  provide  good  fits  (>90%  match)  for  the  primary  polarizability  only. 
We  also  identified  a  subset  of  anomalies  that  produced  low  SNR  data.  We  used  point-to-point 
jitter  for  the  mid- time  channels  (0.5  -  2.0  ms)  in  the  primary  polarizability  to  identify  lower  SNR 
sources.  These  were  flagged  for  review  by  the  analyst  during  the  final  QC  stage. 

We  assigned  non-TOI  rankings  to  anomalies  based  on  symmetry  features  (e.g.,  ratio  of 
secondary  to  tertiary  polarizabilities),  decay  features  (e.g.,  ratio  of  late  time  to  early  time  for 
primary  polarizability),  and  size  features  (e.g.,  primary  polarizability  sum  of  all  time  gates). 
Lowest  confidence  non-TOI  rankings  were  assigned  to  anomalies  showing  good  symmetry,  large 
size,  and  long  decay  features.  Highest  confidence  non-TOI  rankings  were  assigned  to  anomalies 
showing  poor  symmetry,  small  size,  and  short  decay  features. 

After  assigning  an  initial  ranking  value  to  each  anomaly,  we  performed  a  final  QC  of  the  list 
during  which  time  the  analyst  reviewed  the  low  SNR  cases.  For  these  cases,  the  statistical 
variability  of  the  model  parameters  was  particularly  useful  for  assigning  a  final  classification 
decision.  Specifically,  the  repeatability  of  the  polarizabilities  enabled  better  characterization  of 
the  noisy  cases.  Figure  25  shows  an  example  of  noisy  polarizabilities  associated  with  a  deep 
25mm.  Although  each  set  of  polarizabilities  is  relatively  noisy,  when  viewed  together  against  a 
library  reference  it  is  easy  to  identify  a  match  to  the  25mm  TOI. 


Figure  25.  Low  SNR  source.  Each  polarizability  (mageuta  curves)  recovered  from  this  particular  auomaly 
source  is  uoisy  as  a  result  of  the  object’s  depth;  however,  plottiug  all  sets  of  polarizabilities  recovered  from 
souudiugs  withiu  the  ROI  agaiust  the  relevaut  libraries  (dark  grey  curves)  makes  it  fairly  easy  to  ideutify  a 
match  to  the  25mm  TOI. 
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After  completing  the  QC  of  each  classification  decision,  we  assigned  a  final  ranking  value  to 
each  anomaly  according  to  the  standardized  UXO  Test  Site  scoring  template 
(http://aec.army.mi1/Portals/3/technology/uxo01b.html).  We  assigned  decision  values  for  each 
anomaly  in  three  stages:  1)  response  stage;  2)  discrimination  stage;  and  3)  classification  stage. 

The  response  stage  corresponds  to  the  basic  detection  capabilities  of  the  sensor.  Values  assigned 
in  the  response  stage  represented  the  sensor  output  response  amplitude.  Values  above  our  pre¬ 
defined  amplitude  threshold  indicated  detections. 

The  discrimination  stage  indicates  whether  an  anomaly  is  likely  ordnance  (O)  or  clutter  (C). 
Confidence  values  assigned  during  this  stage  indicate  the  probability  that  an  anomaly  is  ordnance 
(higher  numbers  correspond  to  high  confidence  ordnance  is  present;  low  numbers  correspond  to 
high  confidence  ordnance  is  not  present).  These  discrimination  confidence  values  determined 
the  overall  ranking  for  the  entire  list.  During  the  discrimination  stage  we  selected  an  operating 
threshold  indicating  our  effective  “stop  dig”  point  (i.e.,  all  ordnance  identified  with  maximum 
clutter  left  in  the  ground).  This  operating  threshold  was  the  point  on  the  list  at  which  we  changed 
our  discrimination  assignment  from  O  to  C. 

During  the  classification  stage  we  indicated  the  predicted  ordnance  caliber  and  type  for  each 
anomaly  identified  as  potential  ordnance.  For  the  grid  data,  we  also  indicated  whether  the  grid 
contained  no  anomaly  (i.e.,  blank  -  B).  Finally,  we  assigned  estimated  depth  and  orientation 
(i.e.,  Euler  angle)  values  to  items  identified  as  ordnance.  Figure  26  shows  an  example  of  the 
scoring  format  used. 


Figure  26.  Example  APG  scoring  template. 

We  generated  separate  lists  for  the  grid  and  open  field  (indirect  fire)  areas  and  submitted  the  final 
lists  to  ATC  for  scoring.  ATC  used  these  lists  to  score  our  detection,  discrimination,  and 
classification  performance  for  each  blind  survey  area. 


White  River  Technologies 
APG  Demonstration  Report 


36 


September  2014 


7.0  PERFORMANCE  ASSESSMENT 


ATC-generated  Receiver  Operating  Characteristic  (ROC)  curves  for  the  two  blind  survey  areas 
are  presented  in  Figure  27  and  Figure  28.  These  curves  show  the  OPTEMA  detection  (response) 
and  discrimination  performance  achieved  in  both  the  blind  grid  area  and  the  indirect  fire  area. 


- Response 

- Discrimination 

Noise  Level 
Threshold 


Figure  27.  OPTEMA  ROC  curves  for  bliud  grid  performauce.  The  orauge  dashed  liue  shows  prohahility  of 
detectiou  vs.  prohahility  of  false  positive  for  raukiug  assigumeuts  based  purely  ou  seusor  respouse  amplitude 
(respouse).  The  blue  solid  liue  shows  probability  of  detectiou  vs.  probability  of  false  positive  for  raukiug 
assigumeuts  based  ou  classificatiou  decisious  (discrimiuatiou).  The  black  solid  liue  iudicates  our  stop  dig 
threshold  (the  poiut  ou  our  list  at  which  we  chauged  assigumeut  from  orduauce  to  clutter).  The  black  dashed 
liue  iudicates  the  seusor  uoise  threshold  (the  threshold  below  which  we  did  uot  assigu  a  detectiou).  These 
ROC  curves  iudicate  that  we  cau  achieve  detectiou  of  100%  of  the  orduauce  with  85%  clutter  rejectiou.  For 
this  aualysis,  our  operatiug  threshold  was  set  with  maximum  (100%)  efiicieucy  (i.e.,  the  poiut  where 
maximum  clutter  rejectiou  cau  be  achieved  with  uo  decrease  iu  detectiou). 
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Figure  28.  OPTEMA  ROC  curves  for  opeu  field  iudirect  fire  performauce.  The  orauge  dashed  hue  shows 
prohahility  of  detectiou  vs.  prohahility  of  false  positive  for  raukiug  assigumeuts  based  purely  ou  seusor 
respouse  amplitude  (respouse).  The  blue  solid  hue  shows  probability  of  detectiou  vs.  probability  of  false 
positive  for  raukiug  assigumeuts  based  ou  classificatiou  decisious  (discrimiuatiou).  The  black  solid  liue 
iudicates  our  stop  dig  threshold  (the  poiut  ou  our  list  at  which  we  chauged  assigumeut  from  orduauce  to 
clutter).  The  black  dashed  liue  iudicates  the  seusor  uoise  threshold  (the  threshold  below  which  we  did  uot 
assigu  a  detectiou).  These  ROC  curves  iudicate  that  we  cau  achieve  detectiou  of  99%  of  the  orduauce  with 
83%  clutter  rejectiou  (based  ou  100%  efficieucy  operatiug  threshold).  For  this  aualysis,  our  operatiug 
threshold  was  set  with  99%  efficieucy. 

Overall,  these  results  indicate  that  we  can  achieve  detection  of  100%  of  the  ordnance  in  the  blind 
grid  while  rejecting  85%  of  the  clutter  (false  positives)  and  we  can  achieve  detection  of  99%  of 
the  ordnance  in  the  indirect  fire  area  while  rejecting  83%  of  the  clutter  (false  positives).  For  the 
blind  grid  analysis  we  selected  our  operating  threshold  at  100%  efficiency  (i.e.,  the  point  where 
maximum  clutter  rejection  can  be  achieved  with  no  loss  in  detection).  For  the  indirect  fire  area 
we  selected  our  operating  threshold  at  99%  efficiency  (slightly  too  aggressive).  The 
performance  results  for  100%  efficiency  thresholds  are  shown  in  Table  5. 
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Table  5.  OPTEMA  Performance  Summary 


SURVEY  AREA 

PERCENTAGE  ORDNANCE 
DETECTED 

CLUTTER  REJECTION 
ACHIEVED 

Blind  Grid 

100% 

85% 

Indirect  Fire  Area 

99% 

83% 

Details  of  the  specifie  performance  objectives  are  presented  in  the  following  subsections. 
Further  information  regarding  the  ATC  analysis  is  located  in  Appendix  B. 


7.1  OBJECTIVE:  DETECTION  OF  ALL  TARGETS  OF  INTEREST 

The  ATC  scores  show  that  the  OPTEMA  detected  all  ordnance  in  the  blind  grid  and  99%  of  the 
ordnance  in  the  indirect  fire  area  surveyed  (approximately  70%  of  the  total  test  area  was  covered 
including  low,  medium,  and  high  density  portions  of  the  site).  Further  analysis  of  the  results 
indicates  that  the  targets  missed  in  the  indirect  fire  area  were  two  105mm  projectiles  buried  in 
the  8D  to  12D  (8  x  diameter  to  12  x  diameter)  depth  range.  We  applied  the  same  detection 
threshold  to  both  the  blind  grid  and  indirect  fire  areas.  This  threshold  was  selected  based  on  the 
calibration  activities,  and  was  set  well  below  the  threshold  required  to  detect  all  105mm  TOIs  in 
the  calibration  area.  We  achieved  our  objective  of  98%  or  better  detection  for  both  areas. 

While  we  did  not  achieve  100%  detection  in  the  open  field  indirect  fire  area,  review  of  other 
UXO  Technology  Demonstration  Site  reports  indicates  that  the  OPTEMA  detection  performance 
compares  favorably  to  those  of  other  EIXO  technologies  demonstrated  in  the  indirect  fire  area  [3, 
4,  5].  Because  ground  truth  is  not  available  for  this  area,  we  can  speculate  that  this  site  offers 
significant  detection  challenges  for  the  demonstration  technologies. 

7.2  OBJECTIVE:  MAXIMIZE  FALSE  POSITIVE  REJECTION 

Applying  the  100%  efficiency  threshold,  the  OPTEMA  enabled  rejection  of  85%  and  83%  of  the 
clutter  in  the  blind  grid  and  indirect  fire  areas,  respectively.  We  detected  a  total  of  92%  of  the 
clutter  in  the  blind  grid  and  86%  of  the  clutter  in  the  indirect  fire  area.  We  achieved  our 
objective  of  60%  or  greater  clutter  rejection  in  both  areas. 

7.3  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  TOIS 

Overall,  we  achieved  93%  correct  classification  by  type  of  the  ordnance  in  the  blind  grid  and 
90%  correct  classification  by  type  of  the  ordnance  in  the  indirect  fire  area.  The  most  challenging 
targets  for  classification  by  type  were  the  105mm  (73%)  in  the  blind  grid  and  the  81mm  (85%)  in 
the  indirect  fire  area.  The  105mm  presented  a  challenge  in  the  blind  grid  because  it  produced 
polarizabilities  similar  to  those  of  the  105mm  artillery,  which  was  also  present  in  the  blind  grid. 
For  lower  SNR  cases,  it  was  difficult  to  distinguish  the  two  types.  Similarly,  for  very  low  SNR 
cases  in  the  open  field  it  was  difficult  to  distinguish  the  81mm  from  the  60mm. 

7.4  OBJECTIVE:  ACCURATE  AND  PRECISE  TARGET  LOCATION  ESTIMATION 

Our  estimated  target  locations  were  based  on  the  average  values  recovered  from  all  soundings  in 
the  ROT  Using  the  mean  value  proved  to  be  an  effective  method  to  localize  the  each  source. 
Mean  error  values  and  standard  deviation  are  reported  in  Table  6  for  each  localization  parameter 
(i.e..  Northing,  Easting,  depth).  For  each  of  these  parameters,  we  were  within  the  objective 
values  of  10cm  for  both  the  mean  and  standard  deviation. 
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Table  6.  Mean  location  error  and  standard  deviation  for  target  location  estimates. 


BLIND  GRID 

INDIRECT  FIRE 

Mean  (cm) 

Standard 
Deviation  (cm) 

Mean  (cm) 

Standard 
Deviation  (cm) 

Northing 

N/A 

N/A 

1.0 

7.0 

Easting 

N/A 

N/A 

-2.0 

6.0 

Depth 

3.2 

5.3 

1.0 

7.0 

7.5  OBJECTIVE:  PRODUCTION  RATE 


Our  objective  production  rate  was  0.5acre/hr  or  greater  for  both  areas.  This  rate  is  based  on  an 
advance  rate  of  2km/h  and  33%  overlap  in  sensor  coverage  for  adjacent  transects.  Our  objective 
rate  also  factors  in  time  for  vehicle  maneuvers  at  the  end  of  each  line  and  periodic  equipment 
checks. 

We  calculated  production  rate  using  the  timestamps  embedded  in  the  data  files.  For  an 
uninterrupted  data  collection  period  (i.e.,  one  in  which  the  only  stoppages  were  for  routine 
actions  such  as  data  downloads,  sensor  calibration,  file  transfers,  or  acquisition  parameter 
adjustments),  we  calculated  the  timestamp  difference  between  the  first  and  last  files  collected. 
We  then  determined  the  total  area  surveyed  using  the  2-D  data  maps  associated  with  those  files. 

For  the  blind  grid,  we  achieved  a  production  rate  of  0.5acre/hr,  which  met  the  performance 
objective.  Our  total  time  in  the  blind  grid  included  two  full  surveys  of  the  area  as  well  as  a  few 
recollected  lines  due  to  data  acquisition  errors.  We  collected  two  full  surveys  in  order  to 
determine  whether  measuring  a  shorter  decay  period  would  significantly  improve  SNR.  For  the 
second  survey,  we  changed  the  decay  time  measured  from  8.3ms  to  2.8ms.  The  shorter  decay 
meant  that  we  could  average  more  samples  in  each  transmit  cycle  to  reduce  noise.  After 
analyzing  the  data,  we  determined  that  the  8.3ms  decay  data  was  slightly  noisier,  but  overall 
provided  significantly  more  information  for  classification.  We  chose  to  stay  with  the  8.3ms 
setting  for  the  indirect  fire  survey. 

We  expected  that  the  production  rate  for  the  indirect  fire  area  would  be  slightly  higher  than  for 
the  blind  grid  because  the  transects  in  the  grid  are  relatively  short  compared  to  those  in  the  open 
field.  Thus,  a  larger  percentage  of  the  total  survey  time  is  spent  maneuvering  the  vehicle  at  the 
end  of  each  line  in  the  grid  survey.  We  found,  however,  that  the  benefits  of  running  longer  lines 
in  the  open  field  were  overcome  by  the  challenges  associated  with  the  wet  ground  conditions. 
During  our  post-survey  analysis  of  the  data  fide  time  stamps,  it  was  difficult  to  isolate  the  down 
time  related  to  the  challenging  conditions.  Almost  every  transect  run  in  the  indirect  fire  area 
encountered  delays  caused  by  the  wet  conditions;  either  because  the  vehicle  got  stuck  or  the 
array  had  to  be  repositioned  around  deep  water  holes.  Consequently,  our  production  rate  in  the 
indirect  fire  area  suffered.  In  these  conditions  we  achieved  a  production  rate  of  0.3acre/hr,  which 
did  not  meet  our  0.5 acre/hr  objective;  however,  we  are  confident  that  under  better  ground 
conditions  an  open  field  survey  could  achieve  a  production  rate  closer  to  0.6  or  0.7acre/hr. 
Additionally,  lessons  learned  during  operation  of  the  platform  in  the  difficult  conditions 
encountered  in  the  open  area  will  increase  efficiency  in  the  future  if  the  platform  is  deployed  in 
similar  conditions. 
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7.6  QUALITATIVE  PERFORMANCE  OBJECTIVES 

From  a  user’s  perspective,  dynamic  classification  survey  procedures  using  a  tow  sensor  such  as 
the  OPTEMA  are  very  similar  to  those  of  standard  DGM  surveys.  Consequently,  many  of  the 
sensor  positioning  requirements  that  create  challenges  during  cued  surveys  are  not  an  issue.  The 
main  operational  requirements  for  the  OPTEMA  include  basic  line  following  in  the  survey  area 
such  that  complete  coverage  is  achieved  and  periodic  (i.e.,  beginning  and  end  of  the  day) 
calibration  routines  to  ensure  instrument  functionality.  Some  of  the  more  onerous  tasks 
associated  with  cued  surveys,  such  as  frequent  background  data  collection  and  extensive 
maneuvering  to  center  the  sensor  over  a  set  of  2-D  coordinates  are  not  required.  From  this 
perspective,  the  OPTEMA  provides  a  straightforward  solution  for  classification-level  surveys 
and  offers  distinct  advantages  over  the  cued  approach. 

The  greatest  operational  challenge  we  faced  during  the  demonstration  was  posed  by  the  site 
conditions.  The  excessive  levels  of  standing  water  in  the  open  field  led  to  frequent  stops  to 
dislodge  the  vehicle  and  reposition  the  array,  slowing  production  significantly.  While  these 
conditions  were  problematic,  we  do  not  believe  these  challenges  were  indicative  of  any  system 
design  flaws.  These  conditions  would  likely  be  difficult  for  any  vehicle-towed  system.  These 
challenges,  therefore,  represent  technology-level  rather  than  system-specific  limitations. 

Aside  from  the  aforementioned  adverse  site  condition  areas,  the  OPTEMA  surveys  were 
conducted  with  a  large  degree  of  efficiency.  All  personnel  operating  the  OPTEMA  also  had 
considerable  experience  conducting  cued  surveys.  Based  on  the  operator  feedback  from  our 
demonstration,  it  is  clear  that  the  dynamic  surveys  are  more  intuitive  and  straightforward  to 
conduct  than  typical  cued  sensor  surveys. 

In  addition  to  ease  of  operation,  one  of  the  greatest  advantages  of  the  dynamic  survey  approach 
over  the  cued  approach  is  the  ability  to  correlate  classification  features  with  2-D  map  features. 
During  our  post-survey  classification  analysis,  we  identified  several  instances  where  data 
collected  in  high  density  areas  created  some  ambiguity  in  the  2-D  data  map;  however,  once  the 
classification  features  were  extracted  from  the  inversion,  the  source  locations  were  easily 
resolved.  High  density  areas  are  often  problematic  for  cued  sensors.  Because  the  initial  DGM 
survey  is  usually  conducted  with  a  low  resolution  sensor,  it  can  be  difficult  to  identify  the 
optimal  location  for  sensor  placement  during  the  follow-up  cued  survey.  The  OPTEMA  surveys 
clearly  demonstrated  the  advantages  of  conducting  both  the  mapping  and  classification  stages 
from  one  data  set. 

Eigure  29  shows  an  example  of  two  objects  in  the  survey  area  that  were  close  enough  to  produce 
overlapping  responses.  Using  the  detection  algorithm  alone  it  was  difficult  to  separate  the 
anomaly  into  separate  responses.  In  this  case,  if  a  cued  sensor  were  to  follow  a  standard  DGM 
sensor,  the  target  picking  analysis  might  direct  the  cued  sensor  to  a  location  between  the  two 
targets  and  fail  to  characterize  them  individually.  Using  the  OPTEMA  data,  however,  it  was 
possible  to  separate  soundings  associated  with  one  object  from  soundings  associated  with  the 
adjacent  object  during  the  classification  analysis.  In  this  case,  one  group  of  soundings  clearly 
indicated  a  TOI  (81mm)  while  the  other  group  of  soundings  indicated  a  large  piece  of  clutter. 
Because  both  objects  were  of  comparable  size  and  emplaced  at  similar  depths  in  close  proximity, 
it  would  be  difficult  to  distinguish  their  locations  for  a  follow-up  survey  using  the  standard  cued 
approach. 
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Figure  29.  LEFT:  OPTEMA  DGM  data  map  showing  the  anomaly  of  interest  circled  in  hlack.  Using  the  2-D 
data,  it  is  difficult  to  resolve  the  location  of  the  sources;  however,  classification  analysis  of  the  data  reveals 
two  distinct  sources.  RIGFIT:  Polarizahilities  (red  and  magenta  curves)  generated  from  classification 
analysis  of  the  data  plotted  against  library  polarizahilities  (dark  grey  curves).  Soundings  from  within  the 
ROI  (circled  in  hlack)  are  inverted  for  classification  features.  One  set  of  soundings  clearly  indicates  the 
presence  of  an  81mm  TOI  (red  polarizahility  curves  corresponding  to  a  source  location  indicated  hy  the  red 
dot  on  the  map)  while  another  set  of  soundings  indicates  a  large  piece  of  clutter  (magenta  polarizahility 
curves  corresponding  to  a  source  location  indicated  hy  the  magenta  dot  on  the  map). 

Another  example  of  the  improved  mapping/classification  feature  correlation  is  presented  in 
Figure  30,  which  shows  data  acquired  over  an  area  containing  a  cluster  of  objects.  In  this  case, 
the  peak  detector  identifies  eight  potential  sources.  If  a  follow-up  cued  sensor  were  used  here, 
the  cued  sensor  would  need  to  be  moved  around  to  eight  different  locations,  requiring  several 
minutes  to  complete.  Using  the  OPTEMA  data  for  classification,  however,  it  is  apparent  that 
there  are  only  three  significant  sources  in  the  area,  all  of  them  similar  clutter  objects. 


Figure  30.  LEFT:  DGM  map  showing  the  ROI  circled  in  hlack.  This  ROI  corresponds  to  a  cluster  of  targets. 
Using  the  threshold  target  picking  algorithm,  the  peak  detector  generates  eight  potential  source  locations  in 
the  ROI.  If  a  follow-up  cued  survey  were  used,  it  would  entail  soundings  at  each  of  these  eight  locations, 
requiring  several  minutes  to  complete.  RIGHT:  Polarizahilities  (red,  blue,  and  magenta  curves)  generated 
from  classification  analysis  of  the  data  plotted  against  library  polarizabilities  (dark  grey  curves).  The 
polarizabilities  are  divided  into  three  groups  (red,  blue,  magenta),  each  group  corresponding  to  the  location 
of  a  distinct  clutter  object  in  the  DGM  map  (red,  blue,  magenta  dots). 
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8.0  COST  ASSESSMENT 

The  primary  cost  benefit  of  the  OPTEMA  is  the  removal  of  the  cued  survey  from  the 
classification  workflow.  The  costs  of  the  one-pass  classification  method  would  be  comparable  to 
those  of  the  remaining  portions  of  the  existing  workflow.  The  OPTEMA  survey  itself  is  very 
similar  to  the  standard  DGM  survey  conducted  as  part  of  production  operations.  The  anticipated 
0.5-0. 7  acre/hr  production  rate  is  similar  to  what  can  be  achieved  in  open  field  environments 
using  an  EM-61  array.  Requirements  for  field  personnel  are  also  similar  to  those  for  standard 
DGM  surveys  (i.e.,  two  technicians  for  vehicle-towed  surveys). 

Regarding  the  costs  associated  with  the  post-survey  data  analysis,  we  believe  that  costs 
associated  with  analysis  of  OPTEMA  data  would  be  comparable  to  the  combined  costs  of  the 
target  picking  and  classification  analysis  stages  currently  required  for  cued  classification.  The 
one-pass  and  cued  methods  both  require  distinct  target  picking  and  target  classification  stages  as 
part  of  the  analysis.  Time  and  personnel  requirements  for  both  methods  are  similar  (i.e.,  review 
of  each  target  by  a  trained  analyst).  Processing  of  OPTEMA  data  is  slightly  more 
computationally  intensive  due  to  the  increased  volume  of  data;  however,  we  believe  any 
associated  costs  due  to  the  higher  data  density  would  be  negligible  since  any  increased  burden  is 
placed  on  processing  hardware,  not  on  personnel. 

A  secondary  cost  benefit  of  the  OPTEMA  may  be  the  improvements  realized  in  data  quality 
control.  Daily  QC  of  cued  data  requires  significant  oversight  by  a  trained  analyst  throughout  the 
duration  of  the  cued  survey.  Because  sensor  placement  is  so  critical  to  the  success  of  a  cued 
survey,  sensor  placement  errors  for  all  anomalies  surveyed  by  a  cued  sensor  must  be  reviewed  on 
a  daily  basis  to  determine  if  cued  reacquisition  is  necessary.  While  OPTEMA  data  require  daily 
QC  to  ensure  instrument  functionality,  there  is  much  less  burden  on  the  analyst  to  determine 
classification  quality  at  this  stage  in  the  survey.  Overall,  the  lesser  requirements  for  analyst  QC 
throughout  the  survey  could  provide  a  significant  cost  reduction  to  the  overall  survey  process. 

8.1  COST  MODEL 

Table  7  provides  a  summary  of  the  cost  elements  associated  with  an  OPTEMA  survey.  Costs  for 
each  of  these  elements  are  compared  to  those  associated  with  a  cued  survey  (i.e.,  combined 
DGM  and  follow-up  cued  survey). 

Table  7.  Cost  requirements  for  OPTEMA  survey. 


Cost  Element 

Data  Tracked  During  Demonstration 

OPTEMA  Survey 
Costs  Compared  to 
Cued  Survey  Costs 

Instrument  cost 

•  Only  one  sensor  required  compared  to  two  systems  for 
cued 

•  Auxiliary  sensors  comparable  to  cued  (i.e.,  RTK  GPS, 
inertial  sensor,  etc.) 

Lower 

Mobilization  and 
demobilization 

•  Mobilization  and  setup  of  only  one  sensor  compared  to 
two  systems  for  cued 

Lower 

Site  preparation 

•  IVS  setup  comparable  for  both  methods 

Equivalent 
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Survey  costs 

•  Production  rate  of  OPTEMA  is  slightly  lower  than 
production  rate  of  DGM  portion  of  cued  survey  (0.5-0. 7 
acre/hr  vs.  >1.0  acre/hr) 

•  No  follow-up  cued  survey  required 

•  Lower  daily  QC  costs  (less  time  required  for  analyst 
review) 

Lower 

Detection  data 
processing  costs 

•  Target  picking  analysis  of  OPTEMA  data  is  comparable  to 
target  picking  analysis  of  DGM  data 

Equivalent 

Discrimination  data 
processing 

•  Classification  analysis  time/resources  for  OPTEMA  data 
is  comparable  to  classification  analysis  time/resources  for 
cued  data 

Equivalent 

8.1.1  Equipment  Costs 

Overall,  equipment  eosts  should  be  signilieantly  lower  for  the  OPTEMA  survey.  The  eued 
method  requires  two  eomplete  geophysieal  systems  (e.g.,  EM-61  towed  array  and  MetalMapper); 
whereas  the  OPTEMA  method  requires  only  one  eomplete  sensor  system.  We  expeet  that  costs 
of  a  commercially  developed  OPTEMA  would  be  comparable  to  those  of  a  MetalMapper  system. 
Auxiliary  sensor  requirements  (i.e.,  navigation  and  orientation  sensors)  are  similar  for  both  cued 
and  OPTEMA  surveys. 

8.1.2  Survey  Costs 

Survey  costs  would  also  be  significantly  lower  for  the  OPTEMA  survey.  While  production  rates 
for  the  OPTEMA  may  be  slightly  lower  than  those  of  standard  DGM  systems  (e.g.,  EM-61 
towed  array),  no  follow-up  cued  survey  is  required.  Cued  survey  costs  can  be  significant  as  they 
are  a  function  of  both  areal  coverage  and  anomaly  density  (by  comparison,  DGM  and  OPTEMA 
survey  costs  are  driven  primarily  by  areal  coverage). 

Mobilization  costs  are  lower  for  the  OPTEMA  as  well  since  transport  and  setup  of  only  one 
system  is  required  instead  of  two  separate  DGM  and  cued  systems.  Additionally,  daily  QC  costs 
could  be  much  lower  for  the  OPTEMA  as  there  is  reduced  dependence  on  the  off-site  analyst  to 
make  classification  quality  decisions  throughout  the  survey. 

8.1.3  Processing  and  Analysis  Costs 

Post-survey  data  processing  and  analysis  costs  would  be  similar  for  both  methods.  The 
OPTEMA  requires  the  same  detection  and  classification  stages;  the  main  difference  is  that  these 
stages  use  the  same  data  set  compared  to  the  cued  approach,  which  uses  two  different  data  sets 
(DGM  and  cued).  Overall,  the  analyst  requirements  are  similar  for  both  methods. 

8.1.4  Overall  Cost  Analysis 

Here  we  provide  an  example  to  demonstrate  the  potential  cost  savings  that  could  be  achieved  by 
implementing  an  OPTEMA  survey.  Consider  a  100  acre  site  with  an  anomaly  density  of  250 
anomalies/acre.  We  can  apply  a  few  basic  cost  assumptions  using  data  from  the  2014  ESTCP 
Spencer  Range  summary  report  [6].  We  assume  combined  field  survey  and  analysis  rates  of 
$1000/acre  for  an  EM-61  DGM  survey  and  $30/anomaly  for  a  MetalMapper  cued  survey.  At  the 
aforementioned  250  anomalies/acre  density,  this  creates  a  total  survey/analysis  rate  of 
$8500/acre  or  $850,000  total  for  the  site. 

To  determine  an  estimated  cost  for  using  the  OPTEMA  at  this  site,  we  need  to  make  a  few 
additional  assumptions.  We  will  assume  the  EM-61  DGM  production  rate  is  2x  higher  than  the 
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OPTEMA  rate  (1.0  acre/hr  vs.  0.5  acre/hr).  We  will  also  assume  the  OPTEMA  analysis  eosts  are 
somewhat  higher  than  the  DGM  analysis  eosts  sinee  the  OPTEMA  analysis  ineludes 
elassifieation  as  well  as  deteetion  (it  should  be  noted,  however,  that  the  longer  2x  survey  time 
should  mean  that  much  of  the  analyst’s  time  over  that  duration  could  be  spent  on  classification, 
thus  limiting  the  increase  in  the  overall  combined  survey/analysis  rate  for  the  OPTEMA). 

To  be  conservative,  we  will  apply  two  gains  to  the  standard  DGM  survey/analysis  rate.  The  first 
gain  is  a  factor  of  2x,  which  accounts  for  the  lower  production  rate.  The  second  gain  is  a  factor 
of  1.5x  to  account  for  any  increases  in  the  data  analysis  costs.  Thus,  our  combined 
survey/analysis  rate  for  the  OPTEMA  would  be: 

$1000/acre  x  (2.0  production  cost  gain)  x  (1.5  analysis  cost  gain)  =  $3000/acre 

Eor  the  aforementioned  site,  we  therefore  have  a  total  cost  of  $300,000  or  an  approximate  65% 
reduction  in  total  survey/analysis  costs  when  compared  to  the  cued  classification  approach.  The 
cost  savings  becomes  even  more  significant  when  anomaly  densities  are  very  high;  for  example 
if  the  anomaly  density  for  the  same  site  were  600  anomalies/acre.  Total  costs  for  the  OPTEMA 
survey  would  increase  due  to  the  classification  analysis  required  for  more  anomalies.  Here  we 
will  apply  an  analysis  gain  factor  of  2.5x: 

SlOOO/acre  x  (2.0  production  cost  gain)  x  (2.5  analysis  cost  gain)  =  $5000/acre 

This  creates  a  total  survey  cost  of  $500,000.  Total  costs  for  the  cued  survey  would  increase 
dramatically,  however,  to  $1,900,000.  In  this  case,  the  OPTEMA  would  provide  an  approximate 
74%  cost  savings. 

8.2  COST  DRIVERS 

While  the  one-pass  detection/classification  method  could  provide  significant  cost  savings  at  any 
site,  the  greatest  returns  for  OPTEMA  deployments  would  be  realized  in  large  open  field  areas 
and  high  density  sites.  Highest  production  rates  would  be  achieved  in  areas  amenable  to  vehicle- 
towed  surveys,  particularly  those  sites  that  allow  for  long,  straight  survey  transects. 
Additionally,  sites  that  contain  very  high  anomaly  densities  would  also  realize  significant  cost 
savings.  Because  cued  survey  costs  scale  considerably  with  anomaly  density  (i.e.,  production 
rates  for  cued  systems  are  generally  given  in  anomalies/hr  rather  than  acres/hr),  high  density 
areas  significantly  drive  up  cued  survey  costs,  but  have  virtually  no  affect  on  OPTEMA 
production. 

An  added  benefit  of  the  OPTEMA  method  for  these  areas  is  the  improved  correlation  between 
classification  features  and  2-D  map  features  that  is  possible  using  one  data  set.  This  feature 
would  not  only  enable  significantly  better  production  rates  in  these  areas;  it  could  also  lead  to 
better  characterization  of  the  survey  area  and  ultimately  higher  confidence  classification 
decisions  by  the  analyst. 
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9.0  IMPLEMENTATION  ISSUES 

Implementation  issues  for  the  OPTEMA  system  can  be  grouped  into  operational  and  analytical 
categories.  Operational  issues  are  those  related  to  field  deployment  of  the  system  (e.g., 
mobilization,  sensor  calibration,  survey  procedures,  etc.).  Analytical  issues  are  those  related  to 
processing,  analysis,  and  interpretation  of  the  data  acquired  during  the  field  deployment. 

Regarding  the  operational  aspects  of  the  technology  implementation,  the  OPTEMA  functions 
similarly  to  the  standard  DGM  systems  currently  used  for  production  (e.g.,  EM-61  arrays).  Erom 
a  deployment  standpoint,  the  OPTEMA  is  essentially  a  mapping  sensor;  therefore,  field 
procedures  follow  those  of  DGM  surveys.  Requirements  such  as  areal  coverage,  transect 
overlap,  and  instrument  functionality  checks  are  similar  to  those  of  the  DGM  arrays.  Some 
minor  modification  to  the  daily  calibration  routines  conducted  in  the  IVS  are  necessary  to  verify 
classification  performance  of  the  sensor;  however,  these  changes  do  not  affect  the  overall  survey 
requirements  in  terms  of  level  of  operator  training  and  time  allocation. 

Analytical  elements  of  the  technology  implementation  include  data  processing  and  analysis 
requirements  for  both  target  picking  and  classification  stages.  To  a  large  extent,  target  picking 
(i.e.,  ROI  generation)  and  classification  processing  (i.e.,  inversion  of  ROI  soundings)  are 
automated.  We  implemented  automated  routines  to  grid,  interpolate,  and  filter  the  2-D  map  data; 
select  ROIs  based  on  peak  detection  threshold  settings;  and  select  soundings  for  inversion  based 
on  ROI  size.  Some  user  input  was  required  for  background  selection;  however,  this  process 
could  be  automated  as  well.  Once  classification  features  were  generated  for  all  anomalies  in  the 
2-D  data  map,  the  remaining  classification  analysis  was  a  combination  of  automated  ranking  and 
final  review  and  QC  by  the  analyst.  Overall,  these  requirements  are  very  similar  to  those 
currently  associated  with  the  target  picking  and  classification  stages  of  a  cued  sensor  survey. 

The  OPTEMA  represents  a  proof-of-concept  realization  of  dynamic  classification.  As  such,  it  is 
not  a  commercially  produced  system;  however,  at  the  component  level  it  incorporates  many  of 
the  hardware  and  software  features  that  are  currently  part  of  field-ready  commercial  systems 
such  as  the  MetalMapper.  Our  objective  for  this  project  was  to  demonstrate  that  it  is  possible  to 
achieve  classification  performance  comparable  to  that  of  a  cued  sensor  while  operating  in  a 
dynamic  or  mapping  survey  mode.  We  accomplished  this  objective  with  focus  on  the 
development  of  dynamic  data  processing  and  classification  methodologies  rather  than  on  optimal 
hardware  design.  To  a  large  extent,  we  selected  off-the-shelf  and  readily  available  components 
and  combined  them  with  a  low  level  of  customized  fabrication  (i.e.,  the  sensor  head  coils  and 
tow  sled)  to  produce  a  proof-of-principle  prototype  that  could  be  deployed  in  the  shortest  amount 
of  time  possible. 

Consequently,  the  OPTEMA  is  a  fully  ruggedized  field-ready  system;  however,  before  wide- 
scale  production-level  deployment  of  a  dynamic  classification  technology  is  realized,  several 
factors  should  be  considered  for  commercialization: 

•  Sensor  head  configuration.  Our  OPTEMA  represents  one  of  many  configurations  that 
may  be  possible  for  achieving  effective  dynamic  classification.  Eor  this  project,  our  main 
design  constraint  was  to  develop  a  system  that  could  provide  consistent  dynamic 
classification  results  across  an  approximate  2  meter  swath.  Our  system  met  this 
requirement;  however,  there  may  be  other  requirements  that  drive  the  design  of  a 
commercial  system.  Eor  example,  modularity  may  be  important;  i.e.,  can  the  system  be 
designed  in  sections  to  make  swath  width  selectable  by  the  user? 
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•  Size,  weight,  and  power  may  be  important  eonsiderations  as  well.  We  developed  this 
system  for  vehiele-towed  operations;  however,  similar  results  may  be  aehieved  by  a 
scaled-down  man-portable  array  for  dynamic  classification  in  more  prohibitive  operating 
environments.  It  may  be  that  vertical  transmitter  coils  would  be  undesirable  for  operation 
in  more  challenging  terrain.  There  are  a  number  of  design  alternatives  that  could  use 
reduced  coil  size  and  horizontal  layouts. 

•  Data  SNR.  The  OPTEMA  has  very  good  detection  capabilities;  however,  for 
classification  it  is  possible  to  further  improve  data  SNR.  As  we  discussed  in  earlier 
sections,  our  OPTEMA  is  limited  by  the  kickback  capacity  of  the  available  electronics.  It 
would  be  possible  to  modify  transmit  driver  components  to  enable  operation  at  higher 
current  levels.  This  effect  could  provide  significant  gains  in  data  SNR,  which  could 
further  improve  classification  performance  for  deeper  targets. 

We  believe  that  dynamic  classification  systems  may  represent  the  next  generation  classification 
sensor.  While  the  open  field  surveys  conducted  under  this  demonstration  provided  a  significant 
gain  in  the  development  of  dynamic  classification  technologies,  it  is  necessary  to  continue 
deployment  of  systems  such  as  the  OPTEMA  in  realistic  field  environments  to  fully  understand 
the  capabilities  of  the  dynamic  classification  technology.  With  continued  field  evaluations  of  the 
OPTEMA,  we  can  refine  the  design  features  that  would  lead  to  a  wide-scale  realization  of  the 
dynamic  classification  technology. 
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Appendix  B:  ATC  Report 

SECTION  4.  TECHNICAL  PERFORMANCE  RESULTS 
4.1  ROC  CURVES  USING  ALL  MUNITIONS  CATEGORIES 

The  probability  of  detection  for  the  response  stage  and  the  discrimination  stage 
(Pddisc)  respective  probability  of  clutter  detection  or  probability  of  false  positive 

within  each  area  are  shown  in  Figures  7  through  12.  The  probabilities  plotted  against 
their  respective  background  alarm  rate  within  each  area  are  shown  in  Figures  13  through  18. 

Both  figures  use  horizontal  lines  to  illustrate  the  performance  of  the  demonstrator  at  two 
demonstrator-specified  points:  at  the  system  noise  level  for  the  response  stage,  representing  the 
point  below  which  targets  are  not  considered  detectable,  and  at  the  demonstrator’s  recommended 
threshold  level  for  the  discrimination  stage,  defining  the  subset  of  targets  the  demonstrator  would 
recommend  digging  based  on  discrimination.  Note  that  all  points  have  been  rounded  to  protect 
the  GT. 


- Response 

- Discrimination 

---  Noise  Level 
Threshold 


Prob  of  False  Positive 


Figure  7.  OPTEMA/towed  blind  grid  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  probability  of  false  positive. 


Not  covered 

Figure  8.  OPTEMA/towed  open  field  (direct  fire)  probability  of  detection  for  response  and 
discrimination  stages  versus  their  respective  probability  of  false  positive. 


White  River  Technologies 
APG  Demonstration  Report 


50 


September  2014 
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- Response 

- Discrimination 

- Noise  Level 

- Threshold 


Prob  of  False  Positive 


Figure  9.  OPTEMA/towed  open  field  (indireet  fire)  probability  of  deteetion  for  response  and 
diserimination  stages  versus  their  respeetive  probability  of  false  positive. 

Note;  Results  for  the  open  field  (indireet  fire)  are  for  a  partial  submission. 

Not  eovered 

Figure  10.  OPTEMA/towed  open  field  (legaey)  probability  of  detection  for  response  and 
discrimination  stages  versus  their  respective  probability  of  false  positive. 


Not  covered 

Figure  1 1 .  OPTEMA/towed  wooded  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  probability  of  false  positive. 


Not  covered 

Figure  12.  OPTEMA/towed  mogul  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  probability  of  false  positive. 
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- Response 

- Discrimination 

- Noise  Level 

- Threshold 


Prob  of  Background  Alarm 


Figure  13.  OPTEMA/towed  blind  grid  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  probability  of  background  alarm. 

Not  covered 

Figure  14.  OPTEMA/towed  open  field  (direct  fire)  probability  of  detection  for  response  and 
discrimination  stages  versus  their  respective  background  alarm  rate. 
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Figure  15.  OPTEMA/towed  open  field  (indirect  fire)  probability  of  detection  for  response  and 
discrimination  stages  versus  their  respective  background  alarm  rate. 

Note;  Results  for  the  open  field  (indirect  fire)  are  for  a  partial  submission. 

Not  covered 

Figure  16.  OPTEMA/towed  open  field  (legacy)  probability  of  detection  for  response  and 
discrimination  stages  versus  their  respective  background  alarm  rate. 


Not  covered 

Figure  17.  OPTEMA/towed  wooded  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  background  alarm  rate. 

Not  covered 

Figure  18.  OPTEMA/towed  mogul  probability  of  detection  for  response  and  discrimination 
stages  versus  their  respective  background  alarm  rate. 
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4,2  PERFORMANCE  SUMMARIES 


Results  for  each  of  the  testing  areas  are  presented  in  Tables  6  (for  labor  requirements,  see 
section  5).  The  response  stage  results  are  derived  from  the  list  of  anomalies  above  the 
demonstrator-provided  noise  level.  The  results  for  the  discrimination  stage  are  derived  from  the 
demonstrator’s  recommended  threshold  for  optimizing  munitions  related  cleanup  by  minimizing 
false  alarm  digs  and  maximizing  munitions  recovery.  The  lower  and  upper  90-percent 
confidence  limits  on  Pd,  Pcd,  and  Pfj,  were  calculated  assuming  that  the  number  of  detections  and 
false  positives  are  binomially  distributed  random  variables. 
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TABLE  6a,  BLIND  GRID  TEST  AREA  RESULTS 


Response  Stage 

Discrimination  Stage 

Munitions'' 

Scores 

Pd'^“:  by  type 

Pd‘*‘“:  by  type 

All  Types 

105-mm 

81/60-mm 

37/25-mm 

All  Types 

105-mm 

81/60-mm 

37/25-mm 

1.00 

1.00 

0.98 

1.00 

1.00 

0.93 

1.00 

1.00 

0.93 

1.00 

1.00 

0.93 

1.00 

1.00 

0.98 

1.00 

1.00 

0.93 

1.00 

1.00 

0.93 

1.00 

1.00 

0.93 

By  Depth^ 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

Clutter 

Scores 

Pcd 

Pfp 

By  Mass 

By  Depth’’ 

All  Mass 

0  to  0.25  kg 

>0.25  to  1  kg 

>1  to  8  kg 

All  Mass 

0  to  0.25  kg 

>0.25  to  1  kg 

>1  to  8  kg 

All  Depth 

0.95 

0.92 

0.88 

0.83 

1.00 

1.00 

0.19 

0.14 

0.10 

0.02 

0.17 

0.70 

0  to  0.15  m 

0.91 

0.83 

1.00 

1.00 

0.13 

0.02 

0.13 

1.00 

0.15  to  0.3  m 

0.94 

0.80 

1.00 

1.00 

0.25 

0.00 

0.43 

0.25 

0.3  to  0.6  m 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Background  Alarm  Rates 

Pbf“:  0.13 

Pb/’*":  0.01 

‘‘In  cells  with  offset  data  entries,  the  numbers  to  the  left  are  the  result  and  the  two  numbers  to  the 
right  are  an  upper  and  lower  90-pereent  eonfidenee  interval  for  an  assumed  binomial 
distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  objeet. 
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TABLE  6b.  OPEN  FIELD  DIRECT  FIRE  TEST  AREA  RESULTS  (not  covered) 


1  Response  Stage 

Discrimination  Stage 

Munitions'' 

Scores 

P/'";  by  type 

All  Types 

105-mm 

81-mm 

1  60-mm  1 

All  Types 

105-mm 

81-mm 

60-mm 

- 

- 

- 

1  1 

1  1 

1 

1 

- 

- 

- 

- 

1  By  Density  \ 

High 

- 

- 

- 

- 

- 

- 

- 

- 

Medium 

- 

- 

- 

- 

- 

- 

- 

- 

Low 

- 

- 

— 

— 

- 

- 

— 

— 

1  By  Depth^  \ 

0  to  4D 

- 

- 

- 

- 

- 

- 

- 

- 

4D  to  8D 

- 

- 

- 

- 

- 

- 

- 

- 

8D  to  12D 

- 

- 

- 

- 

- 

- 

- 

- 

Clutter 

Scores 

Pcd 

PfP 

1  By  Mass  \ 

By  Depth^ 

All  Mass 

0  to  0.25  kg 

>0.25  to 

1  kg 

>1  to  8  kg 

All  Mass 

0  to  0.25  kg 

>0.25  to 

1  kg 

>1  to  8  kg 

All  Depth 

- 

- 

- 

- 

- 

- 

- 

- 

0  to  0.15  m 

- 

- 

— 

— 

- 

- 

— 

— 

0.15  to  03  m 

- 

- 

- 

- 

- 

- 

- 

- 

0.3  to  0.6  m 

- 

- 

- 

- 

- 

- 

- 

- 

Background  Alarm  Rates 

BAR'^“:  -  1  BAR‘‘'“:  -- 

Groups 

Found 

- 

- 

Identified 

- 

- 

Coverage 

- 

- 

‘‘In  cells  with  offset  data  entries,  the  numbers  to  the  left  are  the  result  and  the  two  numbers  to  the 
right  are  an  upper  and  lower  90-pereent  confidenee  interval  for  an  assumed  binomial 
distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  object. 
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TABLE  6c.  OPEN  FIELD  INDIRECT  FIRE  TEST  AREA  RESULTS 


1  Response  Stage 

Discrimination  Stage 

Munitions'' 

Scores 

P/'";  by  type 

All  Types 

105-mm 

81-mm 

60-mm 

All  Types 

105-mm 

81-mm 

60-mm 

1.00 

0.99 

0.96 

0.99 

0.96 

0.88 

1.00 

1.00 

0.95 

1.00 

1.00 

0.95 

0.99 

0.97 

0.94 

0.99 

0.96 

0.88 

0.99 

0.96 

0.89 

1.00 

1.00 

0.95 

1  By  Density  \ 

High 

0.99 

0.96 

1.00 

1.00 

0.99 

0.96 

1.00 

1.00 

Medium 

1.00 

0.00 

0.00 

1.00 

1.00 

0.00 

0.00 

1.00 

Low 

0.99 

0.95 

1.00 

1.00 

0.96 

0.95 

0.93 

1.00 

1  By  Depth^  \ 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

0.96 

1.00 

0.92 

1.00 

8D  to  12D 

0.85 

0.33 

1.00 

1.00 

0.85 

0.33 

1.00 

1.00 

Clutter 

Scores 

Pcd 

PfP 

1  By  Mass  \ 

By  Depth^ 

All  Mass 

0  to  0.25  kg 

>0.25  to 

1  kg 

>1  to  8  kg 

All  Mass 

0  to  0.25  kg 

>0.25  to 

1  kg 

>1  to  8  kg 

All  Depth 

0.89 

0.86 

0.82 

0.76 

0.98 

0.93 

0.09 

0.07 

0.05 

0.04 

0.05 

0.24 

0  to  0.15  m 

0.87 

0.79 

0.97 

0.93 

0.06 

0.04 

0.05 

0.27 

0.15  to  03  m 

0.73 

0.42 

1.00 

0.92 

0.10 

0.00 

0.00 

0.25 

0.3  to  0.6  m 

1.00 

0.00 

1.00 

1.00 

0.00 

0.00 

0.00 

0.00 

Background  Alarm  Rates 

BAR"“:  0.35  |  BAR"*":  0.01 

Groups 

Found 

1.00 

0.84 

Identified 

0.28 

0.08 

Coverage 

0.65 

0.45 

‘‘In  cells  with  offset  data  entries,  the  numbers  to  the  left  are  the  result  and  the  two  numbers  to  the 
right  are  an  upper  and  lower  90-pereent  confidenee  interval  for  an  assumed  binomial 
distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  objeet. 


Note:  Results  for  the  open  field  (indirect  fire)  are  for  a  partial  submission. 
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TABLE  6d.  OPEN  FIELD  LEGACY  TEST  AREA  RESULTS  (not  covered) 


1  Response  Stage 

Discrimination  Stage 

Munitions'* 

Scores 

by  type 

All  Types 

Small 

Medium 

Large 

All  Types 

Small 

Medium 

Large 

- 

- 

- 

- 

- 

- 

- 

- 

1  By  Depth^  \ 

0  to  4D 

— 

— 

- 

— 

- 

— 

— 

- 

4D  to  8D 

- 

- 

- 

- 

- 

- 

- 

- 

8D  to  12D 

- 

- 

- 

- 

- 

- 

- 

- 

>  12D 

- 

- 

- 

- 

- 

- 

- 

- 

Clutter 

Scores 

Pci 

PfP 

1  By  Mass  \ 

By  Depth^ 

All 

Mass 

Oto 

035  kg 

>0.25  to 

1  kg 

>1  to 

10  kg 

>10  kg 

All 

Mass 

Oto 

035  kg 

>0.25  to 

1  kg 

>1  to 

8  kg 

<  10kg 

All  Depth 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0  to  0.15  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.15  to  03  m 

— 

- 

- 

- 

— 

— 

— 

— 

- 

- 

0.3  to  0.6  m 

— 

- 

- 

- 

— 

— 

— 

— 

- 

- 

>  0.6  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

Background  Alarm  Rates 

BAR*“:  1  BAR‘““: 

Groups 

Found 

- 

- 

Identifled 

- 

- 

“The  two  numbers  to  the  right  of  the  all  types  munitions  result  are  an  upper  and  lower  90-pereent 
confidenee  interval  for  an  assumed  binomial  distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  object. 
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TABLE  6e.  WOODED  TEST  AREA  RESULTS  (not  covered) 


1  Response  Stage 

Discrimination  Stage 

Munitions'* 

Scores 

P/'*;  by  type 

All  Types 

Small 

Medium 

Large 

All  Types 

Small 

Medium 

Large 

- 

- 

- 

- 

- 

- 

- 

- 

1  By  Depth^  \ 

0  to  4D 

— 

— 

- 

- 

— 

- 

— 

- 

4D  to  8D 

- 

- 

- 

- 

- 

- 

- 

- 

8D  to  12D 

- 

- 

- 

- 

- 

- 

- 

- 

>  12D 

- 

- 

- 

- 

- 

- 

- 

- 

Clutter 

Scores 

Pc„ 

PfP 

1  By  Mass  \ 

By  Depth^ 

All 

Mass 

■oBaM 

>0.25  to 

1  kg 

>1  to 

10  kg 

>10  kg 

All 

Mass 

Oto 

035  kg 

>0.25  to 

1  kg 

>1  to 

8  kg 

<  10kg 

All  Depth 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0  to  0.15  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.15  to  03  m 

— 

- 

- 

— 

— 

- 

— 

- 

- 

- 

0.3  to  0.6  m 

— 

- 

- 

— 

— 

- 

— 

- 

- 

- 

>  0.6  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

Background  Alarm  Rates 

BAR*“:  1  BAR"^: 

Groups 

Found 

- 

- 

Identifled 

— 

- 

“The  two  numbers  to  the  right  of  the  all  types  munitions  result  are  an  upper  and  lower  90-pereent 
confidenee  interval  for  an  assumed  binomial  distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  object. 
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TABLE  6f.  MOGUL  TEST  AREA  RESULTS  (not  covered) 


1  Response  Stage 

Discrimination  Stage 

Munitions’* 

Scores 

by  type 

All  Types 

Small 

Medium 

Large 

All  Types 

Small 

Medium 

Large 

- 

- 

- 

- 

- 

- 

- 

- 

1  By  Depth^  \ 

0  to  4D 

— 

— 

- 

— 

- 

— 

— 

- 

4D  to  8D 

- 

- 

- 

- 

- 

- 

- 

- 

8D  to  12D 

- 

- 

- 

- 

- 

- 

- 

- 

>  12D 

- 

- 

- 

- 

- 

- 

- 

- 

Clutter 

Scores 

Pci 

PfP 

1  By  Mass  \ 

By  Depth^ 

All 

Mass 

Oto 

035  kg 

>0.25  to 

1  kg 

>1  to 

10  kg 

>10  kg 

All 

Mass 

Oto 

035  kg 

>0.25  to 

1  kg 

>1  to 

8  kg 

<  10kg 

All  Depth 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0  to  0.15  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

0.15  to  03  m 

— 

- 

- 

- 

— 

— 

— 

— 

- 

- 

0.3  to  0.6  m 

— 

- 

- 

- 

— 

— 

— 

— 

- 

- 

>  0.6  m 

- 

- 

- 

- 

- 

- 

- 

- 

- 

- 

Background  Alarm  Rates 

BAR*“:  1  BAR‘““: 

Groups 

Found 

- 

- 

Identifled 

- 

- 

“The  two  numbers  to  the  right  of  the  all  types  munitions  result  are  an  upper  and  lower  90-pereent 
confidenee  interval  for  an  assumed  binomial  distribution. 

’’All  depths  are  measured  to  the  eenter  of  the  object. 


4,3  EFFICIENCY,  REJECTION  RATES,  AND  TYPE  CLASSIFICATION 

Efficiency  and  rejection  rates  are  calculated  to  quantify  the  discrimination  ability  at 
specific  points  of  interest  on  the  ROC  curve:  (1)  at  the  point  where  no  decrease  in  Pd  is  suffered 
(i.e.,  the  efficiency  is  by  definition  equal  to  one)  and  (2)  at  the  operator  selected  threshold. 

These  values  are  presented  in  Tables  7a  through  7d. 
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TABLE  7a,  BLIND  GRID  EFFICIENCY  AND 
REJECTION  RATES 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

1.00 

0.85 

0.92 

With  No  Loss  of  Pd 

1.00 

0.85 

0.92 

TABLE  7b.  OPEN  FIELD  (DIRECT)  EFFICIENCY 
AND  REJECTION  RATES  (not  covered) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

— 

— 

— 

With  No  Loss  of  Pd 

- 

- 

- 

TABLE  7c.  OPEN  FIELD  (INDIRECT)  EFFICIENCY  AND 
REJECTION  RATES  (not  covered) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

0.99 

0.92 

0.97 

With  No  Loss  of  Pd 

1.00 

0.83 

0.90 

Note:  Results  for  the  open  field  (indireet  fire)  are  for  a  partial  submission. 


TABLE  7d.  OPEN  FIELD  (LEGACY)  EFFICIENCY  AND 
REJECTION  RATES  (not  covered) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

— 

— 

— 

With  No  Loss  of  Pd 

-- 

- 

- 

TABLE  7e.  WOODED  EFFICIENCY  AND  REJECTION  RATES 

(not  covered) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

- 

— 

— 

With  No  Loss  of  Pd 

-- 

- 

- 
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TABLE  7f.  MOGUL  EFFICIENCY  AND  REJECTION  RATES  (not  covered) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

— 

— 

— 

With  No  Loss  of  Pd 

- 

- 

- 

At  the  demonstrator’s  reeommended  setting,  the  munitions  items  that  were  detected  and 
correctly  discriminated  were  further  scored  on  whether  their  correct  type  could  be  identified 
(table  8a  through  8f).  Correct  type  examples  include  20-mm  projectile,  105-mm  HEAT 
projectile,  and  2. 75 -inch  Rocket.  A  list  of  the  standard  type  declaration  required  for  each 
munitions  item  was  provided  to  demonstrators  prior  to  testing.  The  standard  types  for  the  three 
example  items  are  20-mmP,  105H,  and  2.75-inch. 


TABLE  8a,  BLIND  GRID  CORRECT  TYPE 
CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS 


Size 

Percentage  Correct 

25mm 

100% 

37mm 

100% 

60mm 

100% 

81mm 

93% 

105mm 

73% 

105  artillery 

93% 

Overall 

93% 

TABLE  8b.  OPEN  FIELD  DIRECT  FIRE 
CORRECT  TYPE  CLASSIFICATION 
OF  TARGETS  CORRECTLY 
DISCRIMINATED  AS 
MUNITIONS  (not  covered) 


Size 

Percentage  Correct 

25mm 

__ 

37mm 

— 

105mm 

— 

Overall 

- 
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TABLE  8c.  OPEN  FIELD  INDIRECT  FIRE 
CORRECT  TYPE  CLASSIFICATION 
OF  TARGETS  CORRECTLY 
DISCRIMINATED  AS 
MUNITIONS 


Size 

Percentage  Correct 

60mm 

88% 

81mm 

85% 

105mm 

96% 

Overall 

90% 

Note:  Results  for  the  open  field  (indireet  fire)  are  for  a  partial  submission. 


TABLE  8d.  OPEN  FIELD  LEGACY  CORRECT 
TYPE  CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS  (not  covered) 


Size 

Percentage  Correct 

Small 

— 

Medium 

— 

Large 

— 

Overall 

- 

TABLE  8e.  WOODED  CORRECT  TYPE 
CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS  (not  covered) 


Size 

Percentage  Correct 

Small 

— 

Medium 

— 

Large 

— 

Overall 

- 
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TABLE  8f.  MOGUL  CORRECT  TYPE 
CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS  (not  covered) 


Size 

Percentage  Correct 

Small 

— 

Medium 

— 

Large 

— 

Overall 

- 

4,4  LOCATION  ACCURACY 

The  mean  loeation  error  and  standard  deviations  appear  in  Tables  9a  through  9f.  These 
ealeulations  are  based  on  average  missed  distanee  for  munitions  eorreetly  identified  during  the 
response  stage.  Depths  are  measured  from  the  eenter  of  the  munitions  to  the  surfaee.  For  the 
blind  grid,  only  depth  errors  are  ealeulated  beeause  (X,  Y)  positions  are  known  to  be  the  eenters 
of  the  grid  square. 


TABLE  9a.  BLIND  GRID  MEAN  LOCATION  ERROR 
AND  STANDARD  DEVIATION 


Mean 

Standard  Deviation 

Northing 

N/A 

N/A 

Easting 

N/A 

N/A 

Depth 

0.032 

0.053 

TABLE  9b.  OPEN  FIELD  DIRECT  FIRE  MEAN 
LOCATION  ERROR  AND  STANDARD 
DEVIATION  (not  covered) 


Mean 

Standard  Deviation 

Northing 

- 

— 

Easting 

— 

— 

Depth 

- 

- 
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TABLE  9c.  OPEN  FIELD  INDIRECT  FIRE  MEAN  LOCATION 
ERROR  AND  STANDARD  DEVIATION 


Mean 

Standard  Deviation 

Northing 

0.01 

0.07 

Easting 

-0.02 

0.06 

Depth 

0.01 

0.07 

Note:  Results  for  the  open  field  (indireet  fire)  are  for  a  partial  submission. 


TABLE  9d.  OPEN  FIELD  LEGACY  MEAN  LOCATION 
ERROR  AND  STANDARD  DEVIATION  (not  covered) 


Mean 

Standard  Deviation 

Northing 

— 

— 

Easting 

— 

— 

Depth 

- 

- 

TABLE  9e.  WOODED  MEAN  LOCATION  ERROR 
AND  STANDARD  DEVIATION  (not  covered) 


Mean 

Standard  Deviation 

Northing 

— 

— 

Easting 

— 

— 

Depth 

- 

- 

TABLE  9f.  MOGUL  MEAN  LOCATION  ERROR 
AND  STANDARD  DEVIATION  (not  covered) 


Mean 

Standard  Deviation 

Northing 

— 

— 

Easting 

— 

— 

Depth 

- 

- 
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